Artificial Intelligence in Rulemaking: From Retrospective Review to Prospective Rules

Recently, we wrote an essay expanding on a report, Algorithmic Tools in Retrospective Review of Agency Rules (“Report”), which one of us (Sharkey) wrote for the Administrative Conference of the United States (ACUS) to assess how federal agencies can use artificial intelligence (AI) to re-assess the costs and benefits of their regulations after those regulations have been issued, in a process called retrospective review. Our essay (and the Report) encourages agencies to build the technical expertise to develop AI tools for retrospective review, to require that those tools be open source and interoperable with other government technology initiatives, and, perhaps most importantly, to consider how such tools demonstrate the potential of AI not only for retrospective but also prospective analysis of rules. Following the Office of Management and Budget’s (OMB) new revisions to Circular A-4—OMB’s guidance on analyzing regulatory impact—we want to elaborate on the lessons that the Report offers for applying AI in regulatory analysis.

At the outset, we acknowledge that AI is not the right tool for every governmental task. The adage, “when all you have is a hammer, every problem starts to look like a nail,” could be recast in modern parlance to describe how someone with a large language model might think all difficulties can be overcome with natural language processing. But one theme of the Report is its exploration of the impressive degree to which AI is suited to assist agencies in retrospective review. We think that this aptitude has important implications for the entire rulemaking process. More specifically, understanding which aspects of retrospective review make it such a strong candidate for enhancement via AI will illustrate characteristics of potential roles that AI may profitably fill in regulatory analysis and prospective rulemaking.

One reason that AI is so apt a tool for retrospective review is that such review, while an essential part of good governance, is performed all too rarely. As part of the field research for the Report, Sharkey asked agency officials whether their agencies would consider incorporating AI into their retrospective review programs. One official responded enthusiastically, “AI is key in retrospective review, because no one wants to do the work, and it’s low priority; so AI is perfect for that.” Hence, AI is useful where agencies face a choice between automating a valuable but low-priority task, or not doing it at all.

A second reason AI is particularly compatible with retrospective review is that many core review tasks can be performed by existing AI technologies. For instance, the General Services Administration and Centers for Medicare and Medicaid Services (CMS) ran a pilot study on a test set of rules which CMS subject matter experts had identified as imposing conflicting requirements. The agencies commissioned a prototype AI tool to perform “cross-domain analysis” on the chosen regulations, and the resulting AI tool successfully uncovered the same inconsistencies which CMS’s experts had found in the rules. Another project at the Department of Health and Human Services (HHS) corrected nearly 100 citations, deleted erroneous terms, and otherwise redressed misspellings and typographical errors in HHS’s rules. These examples show that agencies did not turn to AI for retrospective review simply because traditional solutions are not feasible or effective, but because it works.

AI is most useful when it fits into, rather than disrupts, existing agency processes. One of the Report’s recommendations—that agencies consider the formats and structures of their rules so that AI-enabled retrospective review tools can more readily and effectively assess agency regulations—was not endorsed by the ACUS official recommendation based on the Report. Although we maintain that there is value in consistently structuring rules to facilitate their comprehension by machines, we recognize that it may not make sense, from the agency’s perspective, to change the way they do high-priority rulemakings for the sake of low-priority retrospective review. Moreover, in the Report’s retrospective review case studies, the most successful AI tools had minimal impact on existing rulemaking workflows. For example, when Congress tasked the Department of Defense (DOD) with creating a framework to manage the sprawling “mountain of policies and requirements” promulgated by DOD’s constituent agencies and military services, DOD decided not to tamper with its rulemaking processes. Instead, it built GAMECHANGER, an AI tool which assembles a structured, unified repository out of DOD’s entire decentralized catalogue of guidance, decreasing the time it takes DOD to respond to policy-related queries “from months to seconds,” and saving almost $11 million in annual costs.

Retrospective review pairs nicely with AI in other, smaller ways. Many core functions of retrospective review are trans-substantive. For example, tools like the Department of Transportation’s (DOT) RegData Dashboard, which estimates the regulatory load of a rule, are applicable to almost every substantive rule the agency makes, providing ample opportunities to offset the cost of the tool’s development. And in low-priority retrospective review processes, an agency might be forgiven for a lack of explainability and transparency, two major shortcomings of AI that have hindered wider governmental adoption. Perhaps this observation guided HHS’s decision not to disclose to the public that an AI-enabled tool which HHS had used in retrospective review was developed by a private contractor—let alone that HHS was using an AI tool for retrospective review at all—until it issued its final “Regulatory Clean Up Initiative” rule, which incorporated the revisions suggested by the AI. Nor did HHS face much criticism, possibly suggesting that agency transparency is less essential in retrospective review, at least in HHS’s relatively uncontroversial case, where AI was used to correct errant spellings and cross-references in HHS rules.

But AI’s use in regulatory analysis would surely be subject to more scrutiny. Consider RegData Dashboard, which quantifies the burdens that DOT’s rules impose on regulated entities. Opacity in that situation would obfuscate how the agency determines the costs of its actions, thereby obscuring any comparison to a rule’s benefits and effectively cutting the public out of the agency’s cost-benefit analysis. DOT’s tool was built using an open-source policy analytics platform developed by the Mercatus Center, which allows for greater public review and engagement than the approach HHS took.

Beyond the question of transparency, other big-picture challenges to the adoption of AI by the government concern the appropriateness of algorithmic retrospective review. At the top of the list is the difficulty agencies face in building internal technological capacity to make and use AI. If agencies are already stretched too thin to perform retrospective review, it seems a tall task to acquire the staff, infrastructure, and organizational resources to automate that review, regardless of the tool’s effectiveness. For this reason, agencies that do have internal technical capacity should consider launching pilot projects on retrospective review and sharing their (preferably open source) tools with other federal agencies, as DOD did with its GAMECHANGER tool.

Having set forth the pros and cons of incorporating AI tools into retrospective review, we are now ready to consider what other situations may be good candidates for algorithmic assistance to agencies. Just as HHS used an AI-enabled tool to spot-check the language and links in its existing rules, AI could quite naturally perform this same function for prospective rules, ensuring that they are well-drafted, consistent, and non-duplicative. Likewise, DOT’s use of its RegData Dashboard to assess the costs imposed by its regulations is just as valuable prospectively as retrospectively. This potential function is particularly important given the new Circular A-4’s emphasis on determining the economic value of uncertain outcomes, and could help an agency choose between alternate regulatory approaches with uncertain results. Moreover, retrospective evaluation of the costs and benefits of regulations presents agencies with a chance to learn whether their prospective predictions about those costs and benefits hit the mark. So, when agencies retrospectively assess regulatory impacts, as DOT does with RegData Dashboard, they gain an opportunity to improve their prospective rulemaking AIs, which can in turn reduce uncertainty as to the effects of regulations those AIs assess in the future.

Our framing so far has centered on the utility of AI tools for tasks like checking citations or identifying conflicting regulatory requirements, without considering whether importing a tool from the retrospective to the prospective context affects the suitability of AI for the task. In reality, both the task and the context matter. The regulatory analyses that accompany rules into the Federal Register are more easily challenged than retrospective analyses, which can alter the priority and the resources an agency is willing to devote to the work. Likewise, the value of fully considering every perspective and providing transparency is a chief concern in prospective rulemaking, where a rule marks a departure from the status quo, as compared to the context of retrospective analysis, where a rule has the merits of familiarity and inertia. Still, we think that similarity of task can compensate for dissimilarity of context in important ways. For example, though we might otherwise be concerned when AI tools that are designed for one context drift or creep into a second context, the overlap in design purpose that occurs when a tool is used for a task identical to the one for which it was designed largely obviates this concern.

The retrospective review case studies show that the use of AI in regulatory review is already upon us, and its use in prospective rulemaking may be imminent. As AI expands into these and other regulatory roles, we recommend studying and applying the lessons of algorithmic retrospective review so that we can make the most of AI’s promise for improving regulatory analysis.

“AI is most useful when it fits into, rather than disrupts, existing agency processes.”