The Precautionary Principle

[Note: This article is part 5 of a series on AI Ethics and Regulation. The previous installment examined why AI risks dominate public discourse even though they remain mostly speculative, whereas software errors are largely ignored even though they have caused tremendous destruction.]

Consider again the claim made at the end of the previous article, that AI might not have caused tremendous damage yet, but we should prepare for it because, in view of what’s at stake, it’s better to be safe than sorry. That point cannot be easily teased apart from AI’s perceived existential risks, which, as we saw, have seeped deeply into the public consciousness courtesy of a sensationalist media hungry for clicks. The better-safe-than-sorry approach to AI regulation derives most of its resonance from the dark sci-fi connotations that the term “AI” has acquired over the years, particularly over the last couple of years. It calls for regulating AI on the basis of the so-called Precautionary Principle, which I will now discuss in depth.

There is, admittedly, no shortage of proverbial wisdom on the importance of precaution: Better safe than sorry; an ounce of prevention is worth a pound of cure; look before you leap; and so on. And, indeed, a precautionary mindset is deeply ingrained in our daily lives. We pay a risk premium to take out fire insurance even though the probability of a home fire is low,¹ because we have a natural and rationally justified aversion to risk, even though the price of the insurance exceeds what a strictly money-maximizing agent would be willing to pay.² For largely the same reasons, many people like to set aside savings in an emergency fund; they endure unpleasant diagnostic tests to guard against statistically unlikely diseases; pay services to regularly back up their files, even though the probability of a catastrophic data loss is small; set up two-factor authentication procedures for an extra layer of security even though their devices are unlikely to get hacked and the two-factor process introduces friction in their login experience; and so forth. And this ubiquity of preventative behavior in ordinary life certainly lends prima facie appeal to the precautionary principle. But surely we can all agree that undergoing a colonoscopy is one thing and regulating nanotechnology is another, so lest we stretch our analogies too far, we need to take a closer look and think more carefully about the underlying issues. Matters of public policy cannot be settled with slogans. We’ll start with a brief overview of the precautionary principle’s journey to date.

The precautionary principle has distinctly European roots.³ It is widely regarded as having been formally introduced for the first time in the early 1970s, when Germany incorporated it “as a binding principle into its first environmental program adopted in 1971” (p. 39).⁴ There are, of course, many incidents in the history of public policy, going back much earlier than the 1970s, that involve some form of precautionary reasoning. In 1854 London, for example, Dr. John Snow famously recommended removing the handle from the Broad Street water pump in Soho, in an attempt to stop a cholera epidemic. He had grounds to believe (though no conclusive evidence) that cholera was caused by contaminated water rather than by airborne transmission of noxious vapors (the “miasma theory” that was prevalent at the time), and reasoned that by removing the handle he would at most inconvenience and anger the citizens of that area if his belief was wrong, whereas, if his theory was right, he would halt the epidemic. As it turned out, he was right. But appeals to an explicitly formulated precautionary principle only started to appear in the early 1970s in Europe, first in Germany and then spreading to neighboring countries, and specifically in the context of environmental law and policy.⁵ As Robert V. Percival wrote in 2005:

The roots of the precautionary principle usually are traced to the concept of vosorgeprinzip developed in Germany during consideration of legislation in the 1970s to prevent air pollution from damaging forests. One translation of vosorgeprinzip into English is “principle of foresight planning,” though that does not adequately capture its true meaning. Vorsorge is “a word that combines notions of foresight and taking care with those of good husbandry and best practice.” It does not demand elimination of risk regardless of its likelihood or the costs entailed in doing so. Rather, vosorge emphasizes the importance of developing mechanisms for detecting risks to human health and the environment so that measures can be taken to prevent harm.

The idea started to spread rapidly after its introduction. EU’s 1992 Maastricht Treaty decrees that European environmental policy “shall be based on the Precautionary Principle." In 1992, the principle became internationally enshrined as “Principle 15” of the famous UN Rio declaration, which proclaimed that

In order to protect the environment, the precautionary approach shall be widely applied by States according to their capabilities. Where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation.

(The second sentence is not easy to grasp given that “lack of full scientific certainty” is the inevitable norm; we’ll revisit this point later.) In that same year, 1992, the principle was incorporated into the Framework Convention on Climate Change (UNFCCC), while also appearing in the Preamble of the 1992 Convention on Biological Diversity (CBD). As Cass Sunstein pointed out in 2005: “Between 1992 and 1999, no fewer than twenty-seven resolutions of the European Parliament explicitly referred to the principle.” Already in 1996, it was noted that “few principles are better ensconced in the law and philosophy of environmentalism than is the precautionary principle, …, a mantra of the green movement.” While the principle was still rooted in environmental policy, its purview had started to expand. A 1998 conference organized by SEHN (the Science and Environment Health Network) couched the precautionary principle as follows: When “an activity raises threats of harm to human health or the environment, precautionary measures should be taken even if some cause and effect relationships are not fully established scientifically. In this context the proponent of the activity, rather than the public, should bear the burden of proof.”

This came to be known as the Wingspread Statement of the Precautionary Principle, which is noteworthy for addressing not just the environment but also human health. There are many technological developments, such as food additives, that might impact human health independently of environmental considerations, so this formulation significantly enlarged the scope of the principle. Of course, a technology might impact human health and the environment, and it might do so in conflicting ways, a possibility that seems overlooked by the disjunction “human health or the environment,” which suggests that the two are always positively correlated in mutually beneficial harmony. (Genetically modified foods, for instance, could be good for the environment but not so for human health.) Another noteworthy contrast between the Wingspread statement and the Rio declaration is that the latter spoke of “serious or irreversible damage”, whereas the former only makes reference to “threats of harm” without saying anything about harm severity. A third important difference is that the Rio declaration explicitly referred to cost-effective measures, but no such criterion appears in the Wingspread statement. Finally, the Wingspread statement marks the first formulation of the precautionary principle that explicitly shifts the burden of proof to the technology advocate. It’s no longer the government or the public who need to show that the technology is problematic; rather, it’s the technology proponent who must show that it is not problematic. However, we’ll see that this shift is only partial, it is not a full reversal. Under any reasonable interpretation of the principle, evidentiary burdens must be shared between proponents and opponents. While most of the burden is indeed shifted to a technology’s proponents, the opponents are not off the hook. They still need to demonstrate that there are credible grounds for concern by meeting the de minimis risk proviso.

In general, the Wingspread statement puts forth a conception of the precautionary principle with four core components, quoted here from the introduction to Protecting Public Health & The Environment: Implementing the Precautionary Principle (pp. 8-9):

1. Preventive action should be taken in advance of scientific proof of causality;

2. the proponent of an activity, rather than the public, should bear the burden of proof of safety;

3. a reasonable range of alternatives, including a no-action alternative (for new activities) should be considered when there may be evidence of harm caused by an activity; and

4. for decision making to be precautionary it must be open, informed, and democratic and must include potentially affected parties.

By 1999, books were already being written on the use of the precautionary principle “in the management of technological risk” in general. By 2001, the New York Times was highlighting the precautionary principle as one of the most important ideas of the year. That same year saw publications claiming that “The Precautionary Principle Also Applies to Public Health Actions.” A 2002 paper out of the Law School of the University of Michigan observed that the precautionary principle appeared “as a standard of international law at the doctrinal level as well as in practice,” with the author noting that “the principle introduces a radical shift in the relationship between science and policymaking” and “opens many possibilities for the regulators.” In 2003, San Francisco formally adopted the precautionary principle as a binding “general City policy.” Writing in 2005, M. Peterson noted that while “the precautionary principle was originally invoked by policy makers for dealing with environmental issues such as global warming, toxic waste disposal and marine pollution”, “in recent years, it has also been suggested that the precautionary principle may as well be applied to medical issues”—not just in public health, but in private medical decision making. (See, for example, this 2005 article on The precautionary principle and medical decision making.) By 2005, the principle had been mentioned “in an increasing number of judicial proceedings, including those in the International Court of Justice, the International Tribunal for the Law of the Sea, and the World Trade Organization Appellate Body, as well as in the courts of many nations, including the supreme courts of India and Canada” (p. 16).

The principle’s magical mystery tour forged ahead at full speed, and by 2010 scholars were observing magical powers indeed: “Sometimes, it [the precautionary principle] appears to be used as “a magic wand”, the invocation of which can justify virtually any policy choice” (p. 2). By 2012 it was taken for granted that “The precautionary principle provides a framework for regulating emerging technologies in general” [my italics]. And by now the principle has been deployed in a tremendous range of policy debates, from animal sentience and e-cigarettes to the recent handling of the Covid-19 pandemic and even foreign policy and international relations. Needless to say, its eventual application to AI was inevitable, particularly in Europe. A 2017 declaration by the European Parliament decreed that AI and robotics should transition from research to commercialization “in compliance with the precautionary principle”. In 2020, the European Parliament declared that citizens’ trust in AI can only be built “in line with the precautionary principle that guides Union legislation and should be at the heart of any regulatory framework for AI.” Precautionary European policies, of course, are not enacted in a vacuum; they have a global impact (what has been dubbed as “the Brussels effect”), particularly in China, though, to a lesser extent, in the U.S. as well. Indeed, voices across the globe have argued that “the precautionary principle might just save us from AI.” In the U.S. the precautionary principle might not be explicitly invoked as often,⁶ but nevertheless it underpins most of the calls for AI regulation in this country. For instance, calls for “legal bans, licensing, and regulatory sandboxing” are all “precautionary tactics,” that is to say, “tactics grounded in the precautionary principle—the idea that technologies should not be used unless proven safe enough” (p. 1371).

The notion that “technologies should not be used unless proven safe enough” reflects the aforementioned shift of the burden of proof, a shift that is widely understood to be part and parcel of the precautionary principle. This shift can certainly be reasonable in some contexts, but it all depends on how strongly the proof requirement is cashed out, because proving that something will not cause harm is proving a negative, and that can easily become a fool’s errand. As Morris put it: “It is not possible to prove that something is harmless any more than it is possible to prove that there are no fairies at the bottom of one’s garden” (p. 10).⁷ While “safe enough” sounds sensible, if vague, other formulations are much stricter. In the case of environmental policy, for instance, at some point in the 1990s a Greenpeace spokesperson on BBC2 Nature (cited in the article The Precautionary Principle in Environmental Policies in the Sage Handbook of Environment and Society, p. 592) stated that:

The precautionary principle is a principle that puts the burden of proof onto the polluter rather than the environment. If a polluter cannot prove that what he is discharging will not damage the environment and will not harm the environment then he simply isn’t allowed to discard that sort of waste.

Nothing is said here either about the gravity of the damage or about the epistemic strength of the desired proof. Naturally, this lack of clarity has sometimes been seized upon by industry lobbyists who have used it as a straw man to argue that the precautionary principle imposes “the impossible demand of establishing the “absence of harm,” with a level of evidence that avoids uncertainty”. In reality, however, no reasonable proponent of the precautionary principle would demand something that cannot be forthcoming. A more plausible risk is a sort of deadlock paralysis, whereby proof of safety cannot be obtained because the activity has been banned or overly circumscribed. A green light could be given if and when the activity is shown to be safe (under reasonable conceptions of safety and evidentiary strength), but how can evidence be gathered in the first place if the activity is not allowed to proceed?⁸

The uneasy procedural questions surrounding the shift of the burden of proof⁹ are symptoms of a more general condition that has attracted recurrent criticism, namely, that there is no one formulation of it—that the principle is inherently vague. As early as 1999, Sandin had already counted 19 substantively different versions of it, while Manson, in an important early logical analysis of the principle in 2002, identified three core dimensions of it, which I will describe shortly, and by gleaning existing usage of the principle from various articles, laws, treaties, protocols, and so on, he listed 7 ways to instantiate the first dimension, 7 for the second, and 6 for the third, for a total of 294 possibilities, any one of which would be vague by itself.

In fact, its deep-seated vagueness is occasionally offered as an explanation of its popularity, as anyone can read anything they want into it. Writing in the mid-1990s, Jordan and O’Riordan noted that “the Precautionary Principle is vague enough to be acknowledged by all governments regardless of how well they protect the environment” (p. 33). They point out:

The Precautionary Principle still has neither a commonly accepted definition nor a set of criteria to guide its implementation. “There is,” Freestone (1991, 30) cogently observes, “a certain paradox in the widespread and rapid adoption of the precautionary principle”: While it is applauded as a “good thing,” no one is quite sure about what it really means, or how it might be implemented.

⋯ Skeptics, however, claim its popularity derives from its vagueness; that it fails to bind anyone to anything or resolve any of the deep dilemmas that characterize modern environmental policy making. (p. 22)

According to the authors, this vagueness is a feature, not a bug:

Like sustainability, the precautionary principle is neither a well defined nor a stable concept. It has become the repository for a jumble of adventurous beliefs that challenge the status quo of political power, ideology and civil rights. Neither concept has much coherence other than is captured by the spirit that is challenging the authority of science, the hegemony of cost-benefit analysis, the powerlessness of victims of environmental abuse, and the unimplemented ethics of intrinsic natural rights and inter-generational equity. It is because the mood of the times needs an organizing idea that the precautionary principle is getting a fair wind.

(The above passage is quoted from a journal version of the same article, p. 191). They view the principle more as a “profoundly radical” political instrument than as a decision-making tool, an instrument whose success lies in its very vagueness, in allowing “scope for extended interpretation” (p. 193). And they predict that “precaution will remain politically potent so long as it continues to be tantalizingly ill-defined and imperfectly translatable into codes of conduct, whilst capturing the emotions of misgiving and guilt” (p. 193).

Most scholars, however, view the pervasive lack of clarity as a serious liability, not as an asset. In an early article on the subject (published in 1991), law professor Daniel Bodansky called the principle “too vague to serve as a regulatory standard” (p. 5). It has not been much better received by philosophers (who have probably made the most earnest attempts to analyze its content and implications). In a largely sympathetic 2006 article, philosopher Stephen M. Gardiner wrote that

[the precautionary principle] remains ill-defined, and its philosophical reputation is low. In particular, the recent literature tends to distinguish between weak and strong versions of the principle, and to regard the first as vacuous and the second as extreme, myopic and irrational.

In 1996, scientists Gray and Brewers wrote that the vagueness of the principle “poses a number of fundamental problems” such as the arbitrariness and inconsistency with which the principle can be (and has been) applied, particularly when coupled with the culture of fear that is endemic to it and which leads to an inordinate emphasis on risks. That culture of fear was promulgated from the outset by thinkers like the late philosopher of environmental ethics Hans Jonas, an early champion of the precautionary principle who called for “an imaginative “heuristics of fear” replacing the former projections of hope,” which “must tell us what is possibly at stake and what we must beware of. The magnitude of those stakes, taken together with the insufficiency of our predictive knowledge, leads to the pragmatic rule to give the prophecy of doom priority over the prophecy of bliss” (The Imperative of Responsibility, p. x).

When the vagueness collides with the “heuristics of fear” and then politics is thrown into the mix for good measure, the result is the perfect fuel for policy decisions that are an embarrassment to the human species. Consider the 2002 food aid fiasco, where the government of Zambia, emulating European attitudes towards GMO foods, turned away tons of food aid donated by the U.S. because it included genetically modified corn (the same kind that millions of Americans consume routinely), explicitly invoking the precautionary principle as their rationale and putting millions of people at risk of starvation.¹⁰ Farther up north, in 2001, Norway’s Food Control Agency invoked the precautionary principle to ban Kellogg’s corn flakes, not because they were genetically modified but rather because they were fortified with vitamins and minerals, which might constitute a health hazard when eaten in uncontrollable amounts. (The ban was subsequently overturned in the courts.)

On this side of the ocean, the poster child of the precautionary principle (and the closest that any US law has ever come to officially implementing the principle) is California’s infamous Proposition 65, originally introduced as a ballot initiative in 1986, aimed at protecting California’s water sources from chemicals known to cause cancer or reproductive defects, and to warn the state’s residents about exposure to such chemicals. Some view the law as unnecessary because water sources were already protected by federal standards enacted by Congress in the mid 1970s¹¹ and enforced by the EPA, but California legislators felt that the EPA was not sufficiently precautionary. Proposition 65 started out by listing 178 chemicals linked to cancer and birth defects and required products containing those substances, even in trace amounts well below federally set thresholds, to carry warning labels. The list of substances had to be updated every year, and by now it has expanded to over 900 entries, some of which are ubiquitous (acrylamide, for instance, added in 1990, occurs whenever any food is toasted, roasted, or fried) and have the most tenuous connections to cancer. As a result, California requires cancer warnings on everything from prune juice to hotels, boats, couches, and even coffee, a situation that has become the butt of international ridicule but has serious consequences insofar as it “fails miserably” in its role as a warning system. Having been completely desensitized to warning labels, the public is apt to pay no attention in those rare cases where alarm may be warranted. Another side effect of Proposition 65 that has not been particularly humorous was the creation of a multi-billion-dollar cottage industry of litigation by predatory law firms that abuse the law’s enforcement mechanism by targeting small businesses with frivolous lawsuits. More will be said about side effects shortly.

It might be protested that any principle can be overzealously applied on occasion; that does not mean that there is something inherently wrong with it. Good ideas can be abused, after all. That might be a valid—if weak—defense in theory. Yet it is difficult to think of any other policy-making principle over the last half century that has been enshrined in internationally recognized documents and has managed to become as widely and enthusiastically adopted while repeatedly giving common sense a black eye.

The dual issue with the precautionary principle is that it overlooks potential benefits to the point where its application might become logically inconsistent, in that it might run afoul of itself. Overly stringent regulation of a new technology can prevent society from realizing the benefits of that technology, which might turn out to have seriously harmful effects. Imagine if a life-saving vaccine had to be tested for decades before it could be approved, so that long-term adverse effects could be conclusively ruled out. Such a policy would risk the deaths of many people, and thus enacting it would contravene the precautionary principle, as long as there was some evidentiary grounds for believing that the vaccine would be effective. So an argument could be made that in such a case the precautionary principle would recommend a course of action p as well as its contradictory opposite, ¬p. Likewise, if we ban or restrict the production of nuclear energy because there is a small risk of devastating accidents, that regulatory intervention will inevitably have side effects, such as shifting energy production modes towards other alternatives, like coal, which could have even worse consequences ultimately. As another example from the nuclear realm, consider the following line of thought for destroying all stockpiles of nuclear weapons (p. 154):

One doesn’t require certainty to take decisions about risks. ⋯ With nuclear winter there would be no second chance. The potential costs are so enormous that it hardly matters for our argument whether the probability that the nuclear winter predictions are basically correct is 10 per cent, 50 per cent, or 90 per cent.

On the flip side, however, a plausible argument can be made that eliminating nuclear weapons (if any such agreement could ever be made and enforced) would scuttle the rational deterrence framework established by the doctrine of mutually assured destruction and would ultimately be detrimental to world peace, and therefore, out of precaution, we should maintain and perhaps even proliferate nuclear arsenals. Or consider the 2003 Iraq war, believed by many to have been launched on the basis of a foreign-policy variant of the precautionary principle. In that case the principle would have just as well advised against the war, on the grounds that it posed blowback risks at least as severe as the alleged risks from Iraq’s WMDs (a combination of risk substitution domestically and risk transformation internationally, see p. 22 of this book). And in retrospect we know that that recommendation would have been the correct one. But the principle would have also recommended a preemptive war, since, as the facts were presented, inaction was catastrophically risky, although we now know that the magnitude of the WMD risk was grossly exaggerated, if not entirely fabricated (which showcases a broader object lesson in political manipulation of alleged risks¹²).

The issue here is conceptually similar to the well-known flaw in Pascal’s wager identified by the “many Gods” objection, whose gist was reportedly captured by Denis Diderot in the 18th century quite succinctly: “An Imam could just as well reason that way.” Pascal’s wager advises precautionary belief in the Christian God on the grounds that the mere possibility of eternal damnation is an unimaginably catastrophic risk. But surely it is possible that Zeus exists and that failure to worship him will result in infinite disutility, so, by the same reasoning, one should worship Zeus; and indeed the same precautionary reasoning would work for any other of the many thousands of cataloged deities vying for worship.¹³ So Pascal’s wager suffers from an indeterminacy that makes it unfit as a decision-making compass: It points in conflicting directions without giving us any rational grounds to prefer one over another. In fact, a similar objection can be made against catastrophic-risk arguments for strict AI regulation. Granting that AGI/ASI might result in human extinction, we may invoke the precautionary principle to ban or shackle AI research. By the same token, we must grant the—quite realistic—possibility that humans will be facing a number of other existential threats in the future, from climate change to new super-contagious lethal viruses and an assortment of other horrors, and that only AGI/ASI will be able to help us survive these; we can then invoke the precautionary principle to promote AI research and development.

Defenders of the precautionary principle sometimes point out that more rigorous approaches to risk management, such as expected utility theory or cost-benefit analysis, might also fail to give us grounds for preferring one action over another if both happen to have the same expected value (utility times probability) or the same costs and benefits. While true, that is quite different from endorsing contradictory actions. If two actions have the same expected value, rational agents will be indifferent to the choice. They can rest at ease knowing that even if they flip a coin, they will still be making a decision that is optimal according to their own analysis. An analytically calculated choice equilibrium stands in sharp contrast to the paralysis that can result from receiving inconsistent recommendations without any discriminating guidance.

The more general issue, above and beyond cases of downright incoherence, is that the precautionary principle fails to engage with tradeoffs and side effects, new problems introduced by (usually) well-intentioned attempts to solve existing problems. Every decision has implications, costs and benefits. A tech company can choose to move fast and keep introducing new functionality at a rapid pace but the inevitable byproduct is tech debt that eventually becomes crippling; a precautionary approach would emphasize robustness and resilience but would incur significant opportunity costs. These are choices that decision makers have to navigate. All too often, people fail to make the right choices because they fail to consider the implications of their decisions. The 1997 book The Logic of Failure, originally written by psychology professor Dietrich Dorner in the late 1980s, describes a number of fascinating experiments where the subjects took on “the role of mayor of a small town or district commissioner in charge of an arid region in Africa” and were asked to make policy decisions to solve “problems arising from such matters as population, resources, unemployment, and crop yields,” while given unlimited virtual funds to use as they saw fit. Computer simulations were then conducted to explore the consequences of those decisions, from a few weeks and months down the road to dozens of years into the future.

The book catalogs many spectacular failures and a few successes. In one experiment, an economist and a physicist set out to tackle the problems of Moros, a fictional “West African tribe of seminomads who wander from one watering hole to another in the Sahel region with their herds of cattle and also raise a little millet” (pp. 2-3). The Moros were beset by high infant mortality, an economy decimated by famines, and dwindling cattle herds ravaged by tsetse flies. The benevolent policy makers gathered and analyzed information, debated policy choices, and, having been given carte blanche, they soon sprang into action, drilling wells to improve irrigation, taking measures against the tsetse flies, improving crop yields by introducing fertilizers and planting different strains of millet, and so on. The results were encouraging—at first. By bringing the problem of tsetse flies under control, the number of cattle increased greatly, while the new wells allowed the Moros to expand their pastureland. But before long, the increased number of hungry cattle led to overgrazing and the destruction of the grass roots. The pastures became barren. The well drilling, while helpful at first, ended up exhausting the remaining groundwater supply. By year 20 there were hardly any cattle left and the Moros were in dire straits. Similar results were obtained in other experiments, e.g., in the fictional Tanaland area, improved medical care and an increase in the food supply caused by the use of artificial fertilizers led to a population increase that eventually resulted in a catastrophic famine. The common denominator was that “existing problems had been solved without thought for the repercussions and the new problems the solutions would create” (pp. 13-14). The few participants who ended up doing well were those who took small, reversible steps,¹⁴ made a conscious effort to consider as many side effects as possible, and were quick to pivot.

The persistent patterns of failure documented in Dorner’s book are observed whenever people deal with highly non-linear dynamic systems featuring a large number of variables with complex interactions. These patterns are painfully familiar to physicians.¹⁵ They are also familiar to software engineers, who often end up introducing new problems by deploying solutions to existing ones, largely due to the same reasons: complex dependencies and interactions of variables, lack of understanding of system dynamics, and failure to think out the changes. But software has some giant advantages that make its management much more tractable: formal programming language theory that provides deductive support for prediction and explanation, the ability to run millions of experiments cheaply and quickly, and engineering tools like regression tests and staged deployment pipelines with automated rollbacks.¹⁶ The real world is infinitely messier.

This doesn’t mean we should refuse to do anything for fear of making things worse. That would be no better than the fear-induced paralysis that can result from strong interpretations of the precautionary principle. But in matters of policy we should not rush to “fix” a problem, let alone nip a speculative risk in the bud, convinced that we are doing a good thing, without first thinking very carefully about the ramifications, taking into account both costs and benefits in accordance with our best science and engineering, and proceeding carefully and in small steps. I’m using the term “cost and benefits” here not necessarily in the strict technocratic sense in which it is usually understood in the intersection of government regulation circles and welfarist economics, but more as a rigorous intellectual tallying exercise intended to create what Sunstein calls “a System II corrective against System I heuristics and biases” (p. 130). If we don’t do that, history shows there is a very good chance we will make things worse, as is almost certainly the case with new AI regulations like New York City’s Local Law 144 (“the bias law”),¹⁷ or as would have certainly been the case with with Idaho’s ill-fated H.B. 118 bill.

A Logical Analysis of the Precautionary Principle

In an influential 2002 analysis, Manson claimed that the core structure common to all formulations of the precautionary principle comprises three parts:

1. a damage condition D that specifies some harmful effect of the activity A in question;

2. an epistemic condition E that “specifies the status of knowledge regarding the causal connections between” the activity A and the harmful effect; and

3. a corrective action (or counteraction) C “that decision makers should take in response to” A.¹⁸ These three components form a “decision tripod”: Whenever there is an activity A that meets the damage condition D and the epistemic condition E is satisfied, corrective C should be applied.

There are many different ways to give legs to this tripod. For instance, the corrective C could be a complete ban of A, or it could be a moratorium, or the encouragement of research aimed at mitigating A’s harmful effect, and so on. The epistemic condition E could be mere possibility or suspicion or some stronger warrant, and the damage condition D could be anything from relatively minor health issues or the pollution of a natural resource all the way to cancer, death, environmental catastrophe, and possibly the extinction of the species. By plugging in different “values” for D, E, and C, we can obtain hundreds or thousands of different variants of the precautionary principle.

Notably absent from Manson’s analysis is the shift of the burden of proof, a key element of the principle’s dialectic.¹⁹ Accordingly, I propose a tetrapod formation by adding a fourth epistemic ingredient:

• A rehabilitation condition R specifying evidentiary obligations whose discharge would restore A to the good graces of the decision makers. After all, if there is no conceivable exculpatory evidence in favor of A (evidence which would establish that A is not, after all, going to trigger the feared damage condition D), then there is no uncertainty in the matter and no debate to be had about precautionary measures. Precaution only enters the picture in the presence of uncertainty regarding the link between A and D. Modulo the caveats mentioned earlier about proving a negative, surely it is possible to conceive of evidence showing that genetically modified foods do not, after all, pose any health risks; or that burning fossil fuels does not cause global warming; and so on. Whether such evidence actually exists is an empirical question. Over time, science might produce a body of knowledge K that renders the conditional probability

  \(P(R\,|\,K)\)

  practically negligible (as is the case in the fossil fuels example, given what we already know about anthropogenic climate change). But if no plausible rehabilitation condition R can even be formulated, then either the matter is settled, in which case precaution is neither here nor there, or worse yet, the reasons given in support of a link between A and D are not empirically falsifiable.

Each of these moving parts raises a host of uneasy questions on its own. For instance, what exactly is to count as a corrective action C? (In the case of global warming, for example, does removing carbon dioxide from the atmosphere count as a proper precautionary intervention?) But by far the greatest source of contentious problems lies in E, the epistemic condition. The key unresolved question is this: What type of evidence and how much of it is required before C becomes imperative? One thing that is not disputed is that some such evidence is necessary. We cannot start enacting regulations on the basis of unfounded suspicions. As Jeroen Hopster puts it in a 2021 paper that defends the PP (an abbreviation for “Precautionary Principle”):

But controversies aside, about one issue all scholars working on the PP seem to agree [on]: precaution is only warranted if a threat of harm constitutes a realistic possibility, rather than a far-fetched fantasy (e.g. Betz, 2010; Carter & Peterson, 2015; Gardiner, Hartzell-Nichols, 2017). We should not take precautionary measures in the face of any dreamed-up catastrophe, however fanciful. Instead, envisioned doom scenarios should surpass a minimal level of plausibility to warrant such measures.

This plausibility requirement goes by different names. Shue (2010, p. 549) designates it as PP’s “anti-paranoia requirement”. Gardiner (2006, pp. 52–53) calls for restricting PP to “realistic outcomes”. Carter and Peterson (2015, p. 8) speak of a “de minimis requirement”, from the legal principle de minimis non curat lex—the court should not concern itself with trifles. The underlying ideas are the same and well-established in the literature on PP: any plausible version of the principle should ignore sufficiently improbable risks, to avoid precautionary paranoia. (pp. 1-2, author’s italics)

It is easy to see why a de minimis requirement is needed. Without it, the precautionary principle could be invoked against an endless array of imaginary risks, such as the conspiratorial belief that 5G causes all sorts of serious health problems, including Covid-19. Clearly, there need to be some rational grounds for taking a risk seriously. And this is why the precautionary principle does not quite reverse the burden of proof. Indeed, the ball is initially in the court of those calling for precautionary measures. They must make a convincing case that their fears are not of the tinfoil variety.

But that is where agreement ends. How exactly to cash out the de minimis requirement remains unclear and tendentious. The discussion can be centered around this question: Does the de minimis requirement need an estimate for the probability

\(P(D\,|\,A)?\)

There are two cases, each of which can be divided into a number of subcases. None of the alternatives bode well for the precautionary principle.

Let’s assume first that the situation amounts to what decision theorists call a state of uncertainty (also known in decision theory as a state of ignorance,²⁰ although philosophers tend to distinguish the two in ways that need not concern us here), where it is either practically impossible to assign meaningful probabilities to the envisaged outcomes, or probabilities may be assigned but there is little reason to place much credence in them. As Rawls puts it in his Theory of Justice, this situation “is one in which a knowledge of likelihoods is impossible, or at best extremely insecure” (Revised Edition, 1999, p. 134; the relevance of Rawls to our discussion will be explained shortly). In such a situation, he says, “it is unreasonable not to be skeptical of probabilistic calculations unless there is no other way out, particularly if the decision is a fundamental one that needs to be justified to others.”

The two subcases here are actually quite different. A state of uncertainty in the classical decision-theoretic sense obtains only under the first alternative, when probabilities are simply unattainable. The case where probabilities have been produced but there is reason to doubt their accuracy is qualitatively different and raises distinct difficulties. We’ll start by examining the first possibility, that in which meaningful probabilities simply cannot be computed, a type of uncertainty described by Keynes as follows (pp. 113-114):

Even the weather is only moderately uncertain. The sense in which I am using the term [“uncertain”] is that in which the prospect of a European war is uncertain, or the price of copper and the rate of interest twenty years hence, or the obsolescence of a new invention, or the position of private wealth-owners in the social system in 1970. About these matters there is no scientific basis on which to form any calculable probability whatever. We simply do not know.²¹

This is exactly the sort of situation in which most proponents of the precautionary principle recommend its application, when the probabilities of the outcomes cannot be meaningfully calculated.²² In those situations, however, it seems very challenging to satisfy the de minimis requirement. If we are fundamentally incapable of assigning meaningful probabilities to outcomes, then how can we decide when a certain outcome is sufficiently plausible? As Hopster puts it, “identifying real possibilities seems impossible in situations where probabilities are entirely absent. After all, if we cannot attach any meaningful probabilities to a given outcome, then it seems that we are also in no position to identify that outcome as a real possibility. Differently put, if we can identify an outcome as a real possibility, then there must be at least some evidence on the basis of which we can justifiably do so. This evidence, in turn, could be described in terms of evidential probability. Hence, in situations where we can identify real possibilities, evidential probabilities cannot be entirely absent.”

There is definitely something to be said for this concern, although it is not true that meaningful probabilities can always be assigned to any purported evidence. There is no widely agreed upon method for assessing “the degree of support” that a body of evidence e lends to a hypothesis h in the general case. Most ways of formalizing that notion hinge on the likelihood

\(P(e\,|\,h).\)

In the special case where e is observed data and h is a scientific hypothesis or a model that logically entails specific constraints on how data is generated or at least what patterns one would expect to find in such data, an objective assessment of the likelihood is much more feasible. But in the general case where h can be a proposition such as “AI will eventually lead to extinction,” e could be taken to be just about anything, and how to compute the likelihood P(e | h) in a way that is not capricious, or indeed whether P(e | h) represents a coherent quantity at all, even for an ideally rational agent, is exceedingly unclear. The demand for numeric probabilities can be finessed by switching to qualitative probabilities or ranking judgments only, but in the absence of a scientific model such attempts ultimately face the same underlying difficulties.

So Hopster’s position can be qualified as follows: Evidence e for a hypothesis h can indeed be assigned a degree of probability, albeit perhaps with imperfect confidence, in the presence of a scientific model couched in the same vocabulary as that used to formulate h and e. If no such model is available, then we have no business trying to assess the probability of a remote outcome. Only clairvoyants can hope to do that. By contrast, in policy debates involving causal mechanisms for which we have credible scientific models, like global warming, genetically modified foods, and so on, meaningful—if indeterminate—probabilities can indeed be computed, with the help of our best empirical science and engineering practices (the latter pertaining, for instance, to the proper construction of simulation models). To quote Hopster again:

Scientists might claim, after integrating different models, that the probability that the West Antarctic marine ice sheet has already become unstable is, say, between 1 and 18%. This practice of assigning probability ranges is widespread in climate science. It is adopted, for instance, by the IPCC, which in its recent assessment reports employs a quantitative ‘likelihood scale’ that explicitly links specific probability ranges with specific statements of likelihood (a 99–100% probability range has the qualifier ‘virtually certain’; a 90–100% probability range is ‘very likely’, and so on; see Mastrandrea et al., 2010).

These are the sort of proper, science-based probabilities that could meet a de minimis requirement. They should be sharply distinguished from pseudo-probabilistic claims like “There is a 5% chance that AI might lead to extinction over the next 20 years” obtained from polls given to authors who have published a few papers in AI conferences, most of them graduate students. When scientists inform a regulatory debate by claiming that there is an “1 in 2.4 million chance” of getting cancer from a certain food additive, they do not arrive at that number by voting. They rely on statistical analyses of experimental data, epidemiological studies, cellular models, biological laws, and carefully made extrapolations followed by rigorous uncertainty analyses. Meaningful probabilities can only be derived from quantitative models informed by scientific theories and techniques, not by the subjective feelings of graduate students (who are exposed to the same barrage of media hype about AI’s risks to boot, and are thus prone to the same availability biases as the general population). More generally, rational policy decisions can only be guided by probability judgments grounded in science and evidence, not obtained by sticking a wet finger in the wind. It may be protested that expert opinions, particularly expert consensus, must surely carry some evidentiary weight, but the short answer is that it does not, unless the propositional content of the consensus satisfies a number of constraints and the causal processes that led to the consensus have a number of desired properties. We will have more to say on consensus as evidence later.

Of course, the absence of probabilities does not mean that we cannot make rational decisions. Classical decision theory does offer some guidance in such situations, although “many questions remain unresolved” as Resnik wrote in his 1987 Choices (p. 21), his authoritative textbook on decision theory, and as continues to be the case today. The only prerequisite is the ability to assign ordinal utilities to the various outcomes, so that we can rank them. This is usually feasible. In that setting, many have observed that the precautionary principle is very similar to the maximin rule, which prescribes the action that yields the maximum minimal utility. Essentially, we look at each action and consider how bad things can possibly get if that action is taken, by finding the outcome in that row of the decision matrix with the smallest utility. Once all of these worst-case outcomes have been identified, we choose the action with the best worst-case outcome. When the stakes are high and a number of other conditions (to be discussed shortly) are satisfied, this ultra-conservative approach might be reasonable. But it has also been the target of trenchant criticism.

The maximin rule was originally introduced by Von Neumann and then articulated by Wald²³ in his 1950 Statistical Decision Functions. It came into prominence when it was singled out by John Rawls in the original (1971) edition of his Theory of Justice as one of the three principles that ought to guide the allocation of social primary goods (“rights and liberties, powers and opportunities, income and wealth”²⁴) in accordance with the original position, a thought experiment in which rational self-interested agents come together to design a societal contract without knowing anything about their own role in that society (not just their job and wealth, but not even their gender, race, intelligence, skills, and so on, although they do have public knowledge about the distribution of such traits), a condition known as being under “the veil of ignorance.”²⁵ The idea is that rational agents placed in those conditions would come to an agreement that maximizes mutual advantages and thus gives rise to a just society. In particular, Rawls set out to show that under these hypothetical circumstances, rational agents would inevitably structure their society on the basis of the three principles that he suggested. (Note that Rawls speaks of two principles because he lumps the second and third into a single two-part principle; but for present purposes it is clearer to distinguish all three.)

The first principle specifies that individuals should be given “the right to the most extensive total system of equal basic liberties compatible with a similar system of liberty for all” (p. 302). The second principle is essentially equality of opportunity. The third and most controversial is the so-called difference principle: “Social and economic inequalities are to be arranged so that they are to the greatest benefit of the least advantaged” (p. 302). This is analogous to the maximin rule: pick the allocation that maximizes the minimum. In particular, if we assume that we are in the original position and we view different methods of allocating primary goods as available actions in the decision matrix, the principle calls for choosing whichever method “raises the floor,” i.e., maximizes the amount of resources available to the least well off. As Rawls writes (pp. 152–153):

The maximin rule tells us to rank alternatives by their worst possible outcomes: we are to adopt the alternative the worst outcome of which is superior to the worst outcomes of the others. The persons in the original position do not, of course, assume that their initial place in society is decided by a malevolent opponent. As I note below, they should not reason from false premises. The veil of ignorance does not violate this idea, since an absence of information is not misinformation. But that the two principles of justice would be chosen if the parties were forced to protect themselves against such a contingency explains the sense in which this conception is the maximin solution. And this analogy suggests that if the original position has been described so that it is rational for the parties to adopt the conservative attitude expressed by this rule, a conclusive argument can indeed be constructed for these principle . Clearly the maximin rule is not, in general, a suitable guide for choices under uncertainty. But it is attractive in situations marked by certain special features.

Rawls tries to derive the difference principle (as a social policy for resource allocation) from the alleged rationality of the maximin rule when a decision maker faces the sort of severe uncertainty that characterizes the veil of ignorance, in addition to a few other conditions.

Rawls explicitly recognizes (e.g., in the cited passage above) that the maximin rule is not in general a rational way to make decisions. Its consistent use would require an extremely conservative mindset with an irrationally inflated degree of risk aversion that would be completely impractical. As the late economist and game theorist John Harsanyi put it in a famous critique of Rawls, “if anybody really acted this way he would soon end up in a mental institution” (p. 595). For example:

Suppose you live in New York City and are offered two jobs at the same time. One is a tedious and badly paid job in New York City itself, while the other is a very interesting and well paid job in Chicago. But the catch is that, if you wanted the Chicago job, you would have to take a plane from New York to Chicago (e.g., because this job would have to be taken up the very next day). Therefore there would be a very small but positive probability that you might be killed in a plane accident.

The maximin principle says that you must evaluate every policy available to you in terms of the worst possibility that can occur to you if you follow that particular policy. Therefore, you have to analyze the situation as follows. If you choose the New York job then the worst (and, indeed, the only) possible outcome will be that you will have a poor job but you will stay alive. (I am assuming that your chances of dying in the near future for reasons other than a plane accident can be taken to be zero.) In contrast, if you choose the Chicago job then the worst possible outcome will be that you may die in a plane accident. Thus, the worst possible outcome in the first case would be much better than the worst possible outcome in the second case. Consequently, if you want to follow the maximin principle then you must choose the New York job. Indeed, you must not choose the Chicago job under any condition—however unlikely you might think a plane accident would be, and however strong your preference might be for the excellent Chicago job.

Clearly, this is a highly irrational conclusion. Surely, if you assign a low enough probability to a plane accident, and if you have a strong enough preference for the Chicago job, then by all means you should take your chances and choose the Chicago job. This is exactly what Bayesian theory would suggest you should do.

Again, Rawls would not disagree, so this is perhaps a bit of a straw man. But Harsanyi pressed the point because it could be argued that a rational agent, even when operating under the constraints of the original position, would not choose to deploy a decision making rule unless it consistently produced optimal results, something that maximin would clearly fail to do.

But let’s delve into the operating constraints of the original position and see to what extent they make a case for the maximin rule. Rawls singles out three “chief features of situations that give plausibility to this unusual rule” (p. 154): (a) the state of deep uncertainty imposed by the veil of ignorance, which precludes the assignment of probabilities to outcomes; (b) the fact that an agent in the original position “has a conception of the good such that he cares very little, if anything, for what he might gain above the minimum stipend that he can, in fact, be sure of by following the maximin rule,” which means that “it is not worthwhile for him to take a chance for the sake of a further advantage, especially when it may turn out that he loses much that is important to him”; and finally, (c) the fact that agents in the original position are facing horrifying possibilities (“grave risks”) that they should very much want to prevent, such as dying by starvation. Rawls argues that, under these three conditions, the use of maximin becomes an inevitable result of rational choice.

There are several issues with this line of reasoning. First, it is not at all clear that ignorance of outcome probabilities would compel a rational agent to disregard probabilities altogether. It is plausible, for example, to suggest that in the absence of available probabilities one should simply assume that all outcomes are equiprobable. This is Bernoulli’s famous principle of insufficient reason, later rebranded by Keynes as “the principle of indifference,”²⁶ which is widely used in machine learning modeling when no prior distribution is known. Then, using a probability of 1/n for each of the n possible outcomes, one can proceed as usual by maximizing expected utility (assuming we have cardinal utility values). This is precisely the route that Harsanyi urges. It is a classically utilitarian approach and thus susceptible to a number of criticisms that we don’t have space to discuss here. But there is certainly an argument to be made that a rational agent with general world knowledge would be aware that most traits, including important genetic traits that correlate with one’s station in life, such as native talents, follow roughly Gaussian distributions, and therefore whatever the agent’s chances might be for falling under the tail left end of a curve, they would be small.²⁷ Accordingly, they might opt for any one of several other resource allocation alternatives that do not necessarily maximize the welfare of the least well off,²⁸ e.g., some version of sufficientarianism-cushioned utilitarianism, which ensures that everyone enjoys access to some minimal level of resources, thereby averting the nightmarish possibilities of naive utilitarianism, after which the system reverts to maximizing utility sums.²⁹ Points (b) and (c) are equally debatable. Rational agents could have a lot to gain by aiming higher than the floor; whether they can be expected to care much for it depends on a host of factors that may not be fixed by the original position.

Returning to precautionary regulation, some scholars have argued that the three conditions singled out by Rawls constitute a “core” precautionary principle that mitigates many of the difficulties associated with the usual vague formulations. Gardiner, for example, has argued that the three conditions identified by Rawls can be understood as sufficient criteria for exercising precaution. Now, regulation is obviously very different from the idealized conditions of a thought experiment like the original position. Nevertheless, if we assume that regulators are operating under deep scientific uncertainty and that the risk they are facing is catastrophic, perhaps we have good enough approximations to Rawl’s conditions. However, Gardiner has nothing to say about the key epistemic issue that any interpretation of the precautionary principle has to resolve: an explication of the de minimis requirement. While he agrees that the Rawlsian conception needs to be supplemented with a de minimis proviso, he simply assumes that some way to cash out that proviso exists and is available. But unless and until that is elaborated, the principle remains inoperable.

At any rate, like any other interpretation that incorporates a de minimis requirement, a Rawlsian version of the precautionary principle would still fail to justify the regulation of AI in order to eliminate existential risks arising from the prospect of ASI, even if we assumed that the de minimis requirement had been satisfied, because regulators would still need to show that every other apparently realistic catastrophe that ASI could help us to avert does not meet that same de minimis requirement. And that would be an impossible task, given how many other nightmare scenarios are realistic and could arguably be averted only with the help of a hypothetical ASI. Unless and until that was done, the principle would still fall prey to the incoherence objections discussed earlier in connection with the many-Gods challenge to Pascal’s wager.³⁰

The case where probabilities are obtainable can also be divided into two subcases, one where the probabilities are known to be accurate and one where the probabilities are not known with confidence. (In practice, of course, the first is a rare limit case of the second, since all probability assessments outside of casinos are uncertain to some extent.³¹ Nonetheless, the distinction simplifies the discussion.) The former case is not particularly interesting, because if highly accurate probability estimates are available, then the relevant science is likely settled and there is little need for a precautionary principle, which is typically understood to apply only when the science is still patchy (when there is “lack of full scientific certainty,” as the ambiguous language of the Rio declaration put it). Insisting on using maximin-style precaution even in the presence of reliable probabilities would require an irrationally high degree of risk aversion.

The uncertainty of a probability estimate p can be modeled by the second-order probability p’ of the proposition that p is correct. Under a subjective interpretation of probabilities, this can be understood as an estimate of our confidence in p. Its practical importance derives from the fact that a high probability of risk is meaningless as a policy-making guide if it is likely incorrect, or if we are very uncertain about it. If we estimate the probability that a certain technology will have cause harm to be high, say 0.4, but our confidence in that estimate is very low, say 0.001, then using the precautionary principle to preemptively regulate the technology is not justified. There are concerns about infinite regress here (don’t we need a third-order probability p’’ about the probability that p’ is correct, and so on?), but perhaps we can brush those aside by appeals to some bounded rationality maxim. The question remains: How is the first-order probability p to be combined with the meta-probability p’ to decide whether the risk in question meets the de minimis threshold?(Another difficult question, of course, is how exactly to set that threshold, but let’s ignore that for now.)

Two general avenues are available. The first is to combine the two numbers p and p’ into a single risk metric q. The obvious suggestion is multiplication:

\(q = p \cdot p'.\)

The closer our confidence in p gets to 1, the closer our metric q will approach p; while the lower our confidence in p, the closer q gets to 0:

\(\lim_{p’ \to 1}{q} = p \;\: \mbox{ and } \;\: \lim_{p’ \to 0}{q} = 0.\)

The problem is this: It could be argued that the more uncertain p is (the lower our confidence in it), the greater the need to apply precaution. I believe this intuition derives from allowing the outcome of the activity A to vary, instead of holding it fixed, as Hopster points out (Section 5). Suppose we are told there is a 95% chance that by 2100 global temperatures will rise between 3.3 and 5.7 degrees Celsius, but our confidence in this prediction is low, say 0.3. The high uncertainty might be cause for greater alarm, because it means that temperatures might in fact rise even more, say by 8 degrees Celsius, which would have devastating consequences and should increase the pressure for precautionary intervention.

However, this does not hold if the effects of the activity A are held constant, as would obviously be the case with extinction (by AI or any other means). When the effects are held constant, greater uncertainty should indeed lower the overall risk, making it less likely to meet the de minimis requirement. In the limit case, when we have zero confidence in our probability estimate for a specific event, such as the extinction of humans, our prediction should not carry any weight at all. So I agree with Hopster that if we indeed have both a probability p that a fixed outcome will be deleterious and a meta-probability about p, then combining both into a single number with multiplication is sensible. There are other ways to use both numbers to arrive at a final plausibility judgment that do not involve collapsing them into a single metric, but there is no need to discuss those here.

There are other problems with the precautionary principle that cannot be covered here due to space limitations. For a flavor, see Peterson’s impossibility results, establishing, e.g., that any minimal version of the precautionary principle is logically inconsistent with a small number of normative conditions characterizing rational decision making (a dominance condition, an Archimedean condition, and the requirement that preferences form a total order),³² and his subsequent argument that the precautionary principle would prevent most clinical trials.

Yet More Problems for Precautionary AI Regulation

Let’s take stock. The only somewhat plausible version of the precautionary principle that can be carved out is essentially a version of maximin under a number of Rawlsian constraints, namely (a) deep epistemic ignorance and (b) the prospect of a catastrophic risk engendered by activity A that would supposedly make it rational to avoid aiming for anything other than averting disaster. Even then, regulators would need to (c) rule out moonshot benefits brought about by A that would be needed to prevent other calamities,³³ and (d) establish that the catastrophic risk in question is realistic (meets a de minimis threshold).

The relevant epistemic ignorance can manifest itself in two ways, either via the total absence of intersubjective probabilities, due to the lack of predictive scientific models, or via significantly insecure probabilistic judgments. In the first case, meeting the de minimis threshold becomes acutely problematic, as it is unclear how to formulate any objective and plausible de minimis criteria under those circumstances. Any “evidence” gleaned under such deep uncertainty that purports to bolster the plausibility of the risk in question will be inherently weak. Nevertheless, I will examine some possibilities in connection with AI, such as expert opinion, hypothetical scenarios, and extrapolations based on historical patterns. Consider now the second case, when there are models available, underwritten by our best science and engineering, that produce rigorous—if uncertain—probabilistic risk judgments. This is by far the most auspicious backdrop for potential applications of the precautionary principle. Difficulties remain even in those cases, e.g., it’s not clear how to specify a threshold or how to compute robust second-order probabilities, but at least in some cases these issues can be finessed.³⁴ The case of global warming might well fall under this category.

In the case of AI regulation, the above means that the precautionary principle is only applicable against existential ASI risks. Other concerns, such as potentially biased decision making that might disadvantage protected groups, cannot be forestalled by enacting regulation on the grounds of the precautionary principle. More conventional risk assessments must be carried out to carefully weigh pros and cons and entertain unintended consequences.

Moreover, we already know that the moonshot proviso (condition (c) above) is deeply problematic in the case of ASI, which could be needed to save humanity from a host of catastrophes within the realm of possibility (ranging from climate change and asteroid collisions to deadly new pandemics and of course AI itself). Thus, any application of the precautionary principle to rein in the development of ASI would be disallowed by the same principle.³⁵

In addition, the level of epistemic ignorance associated with AI’s existential risks is of the first kind discussed above: There are no scientific models that can be used to estimate intersubjective probabilities for the envisaged catastrophes. In fact, the absence of such models is intrinsic in this setting. It’s not just that we don’t have them yet, but we might soon get them as scientists continue their work. We’re not talking about learning more about the impact of certain chemicals on human physiology, or learning more about global climate patterns. AI’s alleged extinction risks are radically different from the typical risks that the precautionary principle is intended to prevent. These differences are substantive and complicate the application of the precautionary principle to AI even further, especially in connection with the de minimis requirement.

The key difference is this: Invocation of the precautionary principle has always hinged on identifying or at least positing a plausible causal mechanism that might produce an unacceptable measure of physical harm. It is this specificity that allows for targeted regulatory measures that can directly address and mitigate the identified risks. In the case of global warming, for instance, we have scientific theories that spell out how physical damage could come about: Industrial activity such as the burning of fossil fuels and deforestation are increasing the concentration of greenhouse gases like carbon dioxide (CO2) in the earth’s atmosphere, acting like a blanket around the planet and leading to global warming and significant changes in weather patterns, sea levels, and ecosystems. There need not be perfect consensus on the theory (there is never perfect consensus in science anyway). But for the de minimis threshold to be met, regulation advocates need to specify plausible causal mechanisms through which the feared harms could materialize.

Likewise, take the case of GM (genetically modified) crops. While the evidence is thin, our science at least suggests concrete causal pathways that could have specific deleterious effects. Examples include allergenicity and gene transfer, among others. The possibility that new proteins introduced into the food supply could act as allergens is based on a well-understood biological process, whereby the immune system reacts to proteins that it identifies as foreign, potentially leading to allergic reactions that could range in severity from mild to fatal (see this article for specific examples). Concerns about horizontal gene transfer (a natural process in which genetic material is transferred from one organism to another without traditional reproduction) are based on the potential of transgenes from GM foods, such as antibiotic resistance genes, which are widely used as selectable markers in GM processes, to transfer to human cells or gut microbiota. As it turns out, all available evidence to date suggests that these risks are extremely low, but at least we know what risks we are discussing and what general causal mechanisms would be implicated if these risks were to materialize. Arguably, that might be enough to consider the risks to be realistic, even in the absence of probabilities.

Likewise for BPA, for nuclear energy, and so on. And in all of these cases it is possible to specify some rehabilitation condition, expressed in the language of physical science, whose satisfaction would successfully defuse the risk. By contrast, consider a chemtrail conspiracy theory asserting that the vapor trails left behind by high-flying planes consist of biological agents that are used for mind control. Those concerned with that “risk” are unable to point to any known causal pathway that would start with the spraying of these alleged agents and end with people on the ground being psychologically controlled by the sinister parties in charge. It would thus be impossible to invoke the precautionary principle, because the de minimis requirement could not be met. And no rehabilitation condition could be proposed, because there are no general scientific facts that could ever falsify belief in the reality of that risk.

The situation is somewhat similar with AI’s alleged existential risks, in that these risks are so speculative that those who portray them as serious possibilities are unable to posit any plausible causal mechanisms that will realize them. To a certain extent, of course, this is because a large part of any such pathway will involve software, whose behavior is not understood by reference to physical laws but rather via general principles of functional design and specific premises regarding the structure of some particular computer program. Consider a program controlling a nuclear reactor, for example, and a catastrophic scenario intended to highlight a possible software malfunction that could lead to a nuclear accident. We cannot dismiss such a scenario simply because there is no general scientific model, couched in the language of physical science, that would underwrite a causal description of the accident end to end. The program malfunction would indeed play a key causal role in the accident, but that malfunction would be neither described nor explained at the level of chemistry or physics, but rather at the level of Dennett’s so-called “design stance,” where we describe, predict, and explain behavior in terms of the program’s functional organization.³⁶

That is a fair point, underscoring the need to adjust our understanding of risk when regulating technology that involves software. In the conventional setting of the precautionary principle, a potential risk brought about by activity A can be (simplistically) viewed as a sequence of n > 0 events

\(e₁, \ldots , e_n\)

that transform an innocuous initial state s₁ into a harmful state s_n + 1 that realizes the damage condition D:

\(s_1 \xrightarrow{e_1} s_2 \xrightarrow{e_2} s_3 \xrightarrow{c_3} \cdots \xrightarrow{e_n} s_{n+1} \)

where each e_i can be understood as a physical event. In that setting, arguing for the plausibility of the transition from s_i to s_i + 1 is done by appeal to science, to nomological principles (along of course with experimental results, mathematical assumptions, etc.). But when computer code is involved, one or more of these events are brought about by the operation of software, and their plausibility cannot be supported by nomological considerations. It can, however, be supported by functional design considerations about the software. In the case of the nuclear reactor program, one could use formal methods, statistical analysis, or simple testing to show that the program might, under some rare circumstances, enter an undesirable state that could result in a serious accident, say, by failing to adjust the cooling system properly, leading to overheating and a possible meltdown. A rehabilitation condition could be an assertion, in some appropriate formalism, that this could not happen, and establishing that condition, at least at the level of an abstract model of the software, could be done by formal methods, or possibly by simulation. More general rehabilitation conditions for other risks could also be formulated, stating, for example, that the program has some desired safety or liveness property.

It is important to note that, relatively speaking, the software-controlled parts of such a chain of events are actually easier to reason about, precisely because such reasoning can be done from the design stance, without the need to descend to the level of physical science. Even when the software is complex and its behavior exhibits nonlinearities and emergent properties, it can still be subjected to rigorous analysis and large-scale experimentation at will.

Of course, this assumes that the alleged risk concerns a particular piece of software, or at least a particular class of computer programs. And that is a big part of the problem with alleged existential risks of AI, which makes them very different from risks that might arise from existing computer programs, such as a particular piece of embedded software (whether in a nuclear reactor, an airplane, or a medical device), or a particular program predicting loan defaults, or one that predicts recidivism risk, and so on. Unlike any of these cases, AI’s existential risk scenarios refer to programs that don’t actually exist. They simply assume the existence of some program that can bring about whatever risk the purveyor of the scenario has dreamed up. These scenarios don’t even bother to specify with what type of program they are concerned. It might be a neural network, or it might be some sort of hybrid connectionist-logical system, or it might be anything at all. To see that this has essentially zero information content, ask for the rehabilitation condition: What is a concrete condition that would need to be satisfied for the program to be deemed acceptable? The question is unanswerable, except in generalities as devoid of content as the arguments for the plausibility of the scenario (“the program would have to be shown to be safe in some appropriate sense”). This means that the proposition which asserts the plausibility of the risk is unfalsifiable, not unlike the chemtrail hypothesis, and cannot be taken seriously.

Moreover, these scenarios feature exceedingly implausible state transitions even for the events that do not involve software. For instance, the author of the ominous TIME piece mentioned earlier imagines the following: “A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing.”³⁷ The software part here involves an AI program emailing a lab, presumably with DNA instructions to manufacture proteins that are somehow malicious. That by itself should be met with incredulity, for a host of obvious reasons, but even the non-AI part of this fantastical chain of events collapses under its own weight, as it radically misrepresents both the existing science underpinning the biotechnology available in such labs and the regulatory processes already in place to control the synthetic production of DNA. Manufacturing genuinely new artificial “life forms” is still fanciful fiction; all we have been able to do so far is reboot pre-existing cell shells with small amounts of synthetic genome material. The leap from producing specific proteins to creating functioning, autonomous life forms, particularly at the macroscopic level, encompasses a vast array of scientific and technological hurdles that we are nowhere close to overcoming. The notion that an AI could email digital instructions to a lab and somehow end up with an army of super-intelligent, autonomous, macroscopic³⁸ living organisms bent on world domination is disconnected from reality.

The creation of microscopic pathogens in such labs, either accidentally or orchestrated by bioterrorists or rogue state actors, is a more credible threat, but that is a threat that should be and is independently, vertically regulated. There are strict measures in place that require labs to screen both the specified DNA sequences and the customers requesting them. The measures might not be perfect, in that, for instance, it is theoretically possible that with successful social engineering bypassing endpoint security, a malicious agent might be able to carry out a DNA injection attack into sequences ordered by benign customers (biologists), and the perverted sequences might evade screening by relying on obfuscation procedures inspired by early malware. This has not happened but it is not impossible. At any rate, such potential vulnerabilities are easily addressed by implementing stricter screening algorithms (something that AI can help with) and enforcing tighter security measures.

So such scenarios flatly fail to discharge the de minimis requirement. They are fictional, not realistic. What about other arguments for the plausibility of existential AI risks? I will consider two types: appeals to the opinions of (some) experts and appeals to the rapid pace of AI progress. Let’s start with the former. While I agree that it is prima facie rational to attach at least some weight to the concerns of pioneers like Hinton or Bengio, those concerns cannot be taken at face value. We must examine the reasoning and evidence behind them. And when that is done, we do not see any new insights. Hinton’s thoughts on the subject are a rehash of the usual clichés, ranging from short-term concerns about disinformation, fake photos, and generic “bad actors” using AI to do generic “bad things;” to longer-term concerns about “killer robots,” which might come to be—through some wholly unspecified causal pathway—because “this stuff could actually get smarter than people.” Bengio’s thoughts on the subject are more rigorous but are also well trodden and have been extensively and trenchantly criticized in the past.

But I would like to say a little more on the subject. Even if there was consensus among experts that AI poses existential threats (and there isn’t, in fact I have found the opposite to be the received view among experts), that consensus would still need to clear a number of epistemic hurdles before it could be granted any evidentiary weight. And the first order of business would be to get clear on what scientific consensus actually is. Scientific consensus refers to scientists’ views about actual matters of science—propositions that are entailed by scientific theories. It should not be confused with mere agreement on some arbitrary subject. If 95% of nuclear physicists prefer the Rolling Stones over the Beatles, or if they believe that there will be a nuclear disaster somewhere over the next 20 years, that is not scientific consensus. It is mere agreement among scientists, in the first case in a matter of subjective taste, and in the second case in a forecasting exercise about whether a highly complex and rare sociotechnical event that is tangentially related to actual nuclear physics might or might not happen years down the line. Contrast that with the scientific consensus on global warming—a consensus about future outcomes of physical systems that is based on a heterogeneous body of scientific models and evidence, including:

• Computer simulations that predict future climate conditions based on different scenarios of greenhouse gas emissions. These models are grounded in fundamental laws of physics, including thermodynamics and fluid dynamics, while incorporating laws of chemistry and biology. They have been extensively tested and validated against observed climate data, consistently showing that as greenhouse gas concentrations increase, so does the global average temperature.

• Extensive empirical evidence from studies of past climates through proxies like ice cores, tree rings, and sediment cores, which demonstrate that the rate and magnitude of the current warming are highly unusual in our planet’s history.

Scientific consensus is attained when the beliefs of scientists converge on the basis of such strong intersubjective evidence and such widely accepted scientific theories and models. And the object of the consensus in such cases is a proposition that is itself scientific, that is, a proposition that is entailed by an appropriate body of scientific theory (in tandem with math, auxiliary assumptions, and empirical data), not a statement to the effect that there will be killer robots, or that there will be a rogue ASI, neither of which is even expressible in the language of physical science.

So unless we’re talking about scientific predictions, and barring trivial cases, expert forecasts about complex, emergent, technogenic, rare (and ASI would not be rare, it would be unprecedented) social-scale events hold no special merit, for a simple reason: No one has a crystal ball. And while Hinton and Bengio have not committed to any specific timeframe in which “killer robots” or a “rogue ASI” might materialize, they have certainly proposed some upper bounds for the emergence of ASI itself, e.g., less than 30 years (“I thought it was 30 to 50 years or even longer away. Obviously, I no longer think that.”).³⁹ We would do well to remember that the path of history is littered with flagrantly wrong “expert predictions.” If anything, the smart money might be to bet against expert forecasts. It was 54 years ago, in 1970, when AI expert Marvin Minsky issued the following prediction with “quite certitude”:

In from three to eight years, we will have a machine with the general intelligence of an average human being. I mean a machine that will be able to read Shakespeare, grease a car, play office politics, tell a joke, have a fight. At that point the machine will begin to educate itself with fantastic speed. In a few months it will be at genius level, and a few months after that, its powers will be incalculable.

Five years before that, another AI founding figure, Herb Simon, had predicted that “machines will be capable, within twenty years, of doing any work a man can do.” Keynes was a giant of economics, but we have seen how his prediction of 15-hour working weeks has fared. Bob Metcalfe is an Internet expert—inventor of the Ethernet, 3Com founder, and Turing award winner. In 1995, he infamously predicted that the Internet would “go spectacularly supernova and, in 1996, catastrophically collapse.” (To his credit, Metcalfe literally ate his words after his prediction failed to come true.) The list of egregiously wrong expert forecasts is long and includes everything from vacuum cleaner experts predicting in 1955 that “Nuclear-powered vacuum cleaners will probably be a reality in 10 years”⁴⁰ to Thomas Edison predicting the perfection of alchemy (“It will be an easy matter to convert a truckload of iron bars into virgin gold”) to Marconi predicting that wireless technology “will make war impossible, because it will make war ridiculous” and Albert Einstein confidently making patently wrong predictions about nuclear energy. (And let’s not even get started on the wrong predictions of Bill Gates and Elon Musk, which by themselves could fill a booklet.) Philip Tetlock, a well-known psychologist at the University of California, has done a good deal of work showing that expert predictions are “no more accurate than random guesses.” In fact, they are roughly as good as chimps throwing darts. It bears repeating: None of these predictions were the province of science. They were just guesses about whether a complex technogenic societal event might or might not occur years out in the future. For that kind of guess, you might as well let a chimp throw a dart.

The last and closely related argument that is worth touching on is the rapid pace of progress in AI and the various “scaling laws” which suggest that pretraining models with increasing amounts of data improves their performance. In the words of Hinton again:

“Look at how it was five years ago and how it is now,” he said of A.I. technology. “Take the difference and propagate it forwards. That’s scary.”

There’s much to unpack here, including a lot of presuppositions about the nature of intelligence and performance evaluation. But whether recent progress represents an opportunity or a threat is an assessment that must be made by serious analysis, not by appeals to emotion, and especially not by fear mongering. To justify precautionary regulation, one must show not simply that the recent rate of progress makes ASI a realistic prospect, which is exceedingly dubious in itself, but something much stronger: that the recent rate of progress makes catastrophic ASI risks a realistic possibility. And this, I believe, cannot be accomplished, partly for reasons I have already discussed.⁴¹

Either way, we should be wary of reading too much into recent progress. Hinton’s quote invites us to propagate it forwards, but a skeptic might draw attention to the hallmark warning of the financial industry: Past performance is not indicative of future results—a ubiquitous disclaimer that all investment materials are legally required to display prominently, though it does little to prevent what is known as this-time-is-different syndrome. Perhaps Russell’s chicken is a more vivid way of making the same point about over-generalizing from limited past data:

Domestic animals expect food when they see the person who usually feeds them. We know that all these rather crude expectations of uniformity are liable to be misleading. The man who has fed the chicken every day throughout its life at last wrings its neck instead, showing that more refined views as to the uniformity of nature would have been useful to the chicken. (p. 35)

Yes, things scale—until they don’t. In the heyday of symbolic AI, expert system technology was being heralded with Turing awards, new expert systems startups were being “formed at the rate of what seemed like one a week,” and it was felt that scale was the only thing keeping us from AGI, a challenge valiantly taken on by the Cyc project of the late Doug Lenat, which aimed to build “one immense knowledge base spanning human consensus reality, to serve as scaffolding for specific clusters of expert knowledge” (p. 188), an effort that would take “three decades” (p. 209), which meant that AGI was “within our grasp” (p. 188). Lenat and Feigenbaum were writing this in 1987. And in 1992, Feigenbaum, a Turing-award winner and pioneer of expert system technology, was writing that “as I … look at the future, the thousands of expert systems that have been done provide massive overwhelming evidence for the soundness of the knowledge-is-power hypothesis.” Clearly, deep learning has taken us much farther than expert systems ever did, at least when it comes to language-based, general-purpose AI (in narrow domains, well-crafted expert systems can still outperform), so perhaps this time really is different. But deep learning has not delivered AGI, let alone ASI. To think that it will do so simply because of recent successes is the sort of inductive hubris that does not age well.

1. About 0.03% according to 2010 data.

2. If your house is worth $400K, then your expected loss is 0.03% ⋅ $400K = $120. If you only cared about maximizing money, an insurance plan priced at anything over $120 would not appeal to you—it would not be a fair bet. $120 is the “actuarially fair premium” (AFP) of the insurance; that’s what an insurance company that didn’t care about making a profit would expect to pay out in order to insure your home. However, you might agree to pay $150 for the insurance because you view the extra money that you are spending above the AFP (which is the so-called “risk premium,” in this case $150 - $120 = $30) as better spent protecting you against a catastrophically life-altering loss of $400K than as sitting idle in your bank account, or even in your 401K.

That’s because money has diminishing marginal utility; generally speaking, the more money we have the better, but the increase in utility is sub-linear, say, logarithmic (a suggestion that was first made by Nicolaus Bernoulli in an attempt to resolve the St. Petersburg paradox). Unless you are either starving, in which case that $30 has a lot more immediate value to you, or extremely wealthy, in which case a loss of $400K might be immaterial, the $30 premium that you pay (a good chunk of which ends up as profit in the pockets of the insurance company) is worth the elimination of the risk.

3. The cliché is that in Europe a product is risky until proven safe, whereas in the U.S. a product is safe until proven risky. The stereotype behind that cliché depicts “Europeans as risk-averse, afraid of the unknown, afraid of new technologies and of global markets, jumping to adopt precautionary regulations against the most remote and speculative risks,” while “Americans are seen as risk-indifferent or even risk-preferring, blustering blithely past risks, confident that new technology and the power of (American) markets will solve every problem and that precaution is a waste of time and a hindrance to progress”, as this 2003 article puts it. There is a kernel of truth in that, but, as usual, the reality is more nuanced. Precaution as an attitude or a general approach is quite common in American law as well, even though this was never quite codified as a general principle; see the 2010 article An American Perspective on the Precautionary Principle and also the 2006 article by MIT’s Nicholas A. Ashford on The Legacy of the Precautionary Principle in US Law.

In certain verticals, in fact, the U.S. might be more precautionary than Europe. Cass Sunstein, for instance, a moderate progressive who was the “regulation czar” of the Obama administration (if a somewhat controversial one due to his championing of cost-benefit analysis, a practice that has become increasingly contentious despite wide bipartisan support), submits that the U.S. “is more precautionary about new medicines than are most European nations.” Indeed, a careful study in the same 2003 article by Jonathan B. Wiener cited above found that “on several prominent examples, including particulate air pollution, mad cow disease in blood, youth violence, and terrorism, it is the United States that is acting in the more precautionary manner.” Even in Europe there used to be significant inter-country variations in how the precautionary principle was understood and applied, although, within the EU anyway, increasing political homogeneity has diminished such discrepancies (though not quite eliminated them, e.g., Sweden’s recent handling of the Covid-19 pandemic was very different from that of other European countries, which were much more aligned with the precautionary principle).

4. Some authors contend that Sweden’s landmark Environmental Protection Act, introduced in 1969, already contained a version of the precautionary principle.

5. Indeed, for most of its life the precautionary principle was strongly associated—and occasionally even conflated—with the legal principle of in dubio pro natura (if in doubt, decide in favor of the environment).

6. Though explicit appeals are not hard to find, e.g., the CEO of Microsoft AI, in a December 2023 article co-authored with Ian Bremmer, urged that “The already widely used precautionary principle needs to be adapted to AI and enshrined in any governance regime” [my italics].

7. A 1993 article in the New Scientist put it this way:

Wittgenstein and Bertrand Russell once sat on a couch together and tried to prove that there was not a hippopotamus in the room. In the end they had to get down on their knees and look under the sofa. ⋯ If Wittgenstein and Russell cannot do it without effort, it is because it cannot be done, and no scientist has ever wasted a minute in the attempt.

8. In the U.S. it is common practice to fluoridate public water (as is the case in many other countries), and the CDC has listed fluoridation as one of the ten greatest public health achievements of the 20th century, owing to its contribution to the dramatic decline in dental cavities. The introduction of the practice in 1945 was limited to two cities only. The U.S. PHS (Public Health Service) wanted to get data from those two cities over a period of 10 years and compare the cavity rates to those of other cities before deciding whether to roll out the policy to the general public. But immediately upon its experimental introduction, “politically conservative groups in Wisconsin protested against adding a toxic chemical to their drinking water, arguing that it was used as rat poison and that involuntary fluoridation amounted to mass medication—a step toward socialism.” (In Kubrick’s Dr. Strangelove, the paranoid Brigadier General Jack D. Ripper confides to a bemused Peter Sellers that “Fluoridation is the most monstrously conceived and dangerous communist plot we have ever had to face.”) In the 1969 book The Politics of Community Conflict: The Fluoridation Decision, cited on p. 42 of the above fascinating article by Mazur, the authors write:

The reputed side effects of fluoridation run from destruction of teeth to liver and kidney trouble, miscarriages, the birth of mongoloid children, and psychological disturbances, including susceptibility to communism and nymphomania. When the public-health officer points out that nearly a tenth of the drinking water in the U.S. has always had traces of fluoride in it without causing ill effect, the critics then charge that fluoridation damages car batteries, rots garden hoses, and kills grass.

However, given that fluoride is in fact an acute poison and that nothing was known at the time about the long-term effects of continuous exposure to even microscopic doses of it, the precautionary principle, had it been available at the time as a decision rule, would have advised against public fluoridation. And had the advice been heeded, the data would have never been generated and the public benefits never materialized, as all of the human data we have on the subject so far is based on retrospective studies that compare, for example, cancer rates in communities before and after water fluoridation (we also have data from animal experiments showing, for instance, that low enough doses do not seem to have adverse effects on mice, but that sort of data could have easily been dismissed as failing to meet the burden of proof on the grounds of physiological and metabolic differences between humans and mice, remaining uncertainty about the long-term effects of chronic exposure, variabilities in human sensitivity, and so on). This is not to say that deciding against fluoridation would have been misguided, only that a strong shift of the burden of proof would have effectively shut down the debate.

9. What constitutes acceptable proof of safety? Who gets to make that decision and how? How grave does the threat of harm need to be to shift the burden?

10. In January 2003 “some 6,000 hungry people in one rural town [in Zambia] overpowered an armed police officer to loot a storehouse filled with U.S. corn that they heard the government was soon going to insist had to be taken out of the country.”

11. The Clean Water Act of 1972 and the Safe Drinking Water Act of 1974.

12. Even when explicit risk analysis is performed (a practice that is generally opposed by proponents of the precautionary principle ), it is still a politically fraught process. The party that gets to estimate the risks can—and often does—put their thumbs on the scale, often in favor of worst-case scenarios. Around the turn of the century, for instance, the EPA would calculate the cancer risk from air pollution as that of a fictional person (dubbed “MaxiMan”) undergoing “continuous exposure for 24 hours per day for 70 years” at the site of maximum possible pollution. This does not make for realistic estimates.

13. I am told that not all of them forbid belief in other deities, but others are less open-minded.

14. These are what Amazon calls “two-way door decisions”—decisions that have “limited and reversible consequences.”

15. The term iatrogenesis, which refers to harm brought about (“genesis”) by medical interventions (“iatro”) was only introduced about a century ago, but physicians have known since antiquity that there is no such thing as a risk-free medical decision.

16. The other crucial difference, of course, is that fixing software bugs is reactive, not proactive—we are not taking action in order to forestall a hypothetical future risk, but to address a clear and present problem.

17. The law was enacted without an iota of evidence that resume-screening algorithms that companies actually use (in very limited ways, as was discussed in a previous article) have any systemic bias problems, or even that they are more biased on average than humans (there are very good reasons to expect the opposite, as was also discussed previously). Interest in this area was stirred largely because of an apocryphal story—which nevertheless received extensive media coverage—about an experimental Amazon algorithm dating back to 2014 that was never actually used by recruiters. The law’s journey illustrates the most egregious pattern of misguided precautionary regulation: The media broadcasts an exaggerated version of a single problematic anecdote, boosting the public’s availability bias and making the perceived risk more familiar and salient; the increased cognitive availability of the “risk” then becomes fertile ground for exploitation by ambitious legislators, federal or local, eager to seize attention and score political points (see pp. 327-328 of this paper for an acute example in the context of national environmental legislation, where President Nixon and Senators Edward Muskie and Henry Jackson engaged in “competitive credit-claiming” in an attempt to “corner the market” for the “credit to be claimed from legislation assuring the public of a cleaner world”). Aided by well-meaning academics and activists who are just as susceptible to System I biases as the general public, the end result is regulatory overreach misaligned with empirical realities. NYC’s final definition of what constitutes an AEDT (“automated employment decision tool”) is so sweeping as to encompass any “computational process” that is “used to substantially assist or replace discretionary decision making,” potentially requiring even keyword-matching scripts to undergo annual bias audits, while “specifying the definitions, standards and outputs for an audit is a herculean task.”

After more than two years of rule making, multiple public hearings, and hundreds of pages of generated commentary, the law is now seen as “an overly hyped-up piece of regulation that isn’t well drafted” and “doesn’t do what its intended purpose states it’s supposed to do,” in addition to causing “anti-competitive” side effects like “discouraging the use of tools that—while they can inadvertently discriminate and should not—probably have legitimate business use cases which are being passed over.” Its defenders insist that the law flopped because it became “toothless” after being “watered down” in the wake of a “combative and reactionary” response from “business interests,” and that it would produce good results if it was only stricter and enforced with more zeal, but that is simply not true. The law was fundamentally misconceived from the start and makes for an object lesson in knee-jerk application of the precautionary principle. Similar laws have already been passed in Illinois and Maryland and are being contemplated in many other places.

18. This is often called a remedy, but I will reserve the letter R for another use.

19. Recall the second point in the list presented earlier that forms the interpretation of the Wingspread statement given in the introduction to Protecting Public Health & The Environment: Implementing the Precautionary Principle.

20. See Peterson’s Introduction to Decision Theory, p. 41.

21. This acknowledgment did not stop Keynes from issuing long-term forecasts. He famously predicted, for instance, that automation and productivity advances in general would lead to a 15-hour working week by 2030. File that under the rich category of spectacularly failed predictions made by brilliant people. (In Keynes’s defense, he made that prediction in 1930, 7 years before writing the cited text.)

22. By “meaningfully” here I mean in a way which ensures calibrated probabilities that correspond to long-run relative frequencies, or at least Bayesian probabilities on which it is possible to attain intersubjective agreement based on objective epistemic criteria (coherence, consistency, argument strength, etc.).

23. Incidentally, perhaps the best-ever illustration of survivorship bias comes from an anecdote from Wald's involvement in World War II as a consultant for the allies.

24. 1971 edition, p. 62.

25. One might wonder how it is logically possible for people to be self-interested agents without having any notion of a self (without knowing anything about themselves), but a general analysis of Rawls’s thought experiment is beyond the scope of this discussion.

26. Referring to the phrase “principle of insufficient reason,” Keynes writes that “this description is clumsy and unsatisfactory, and, if it is justifiable to break away from tradition, I prefer to call it the principle of indifference” (p. 44). Refer to Chapter 12 of Bas C. van Fraassen’s Laws and Symmetry for a fascinating discussion of the principle’s history and implications.

27. Rawls defines the “least advantaged” segment of the population as those whose “income and wealth” is less than half of the median.

28. Modulo the liberty and equality of opportunity constraints, which take precedence over the difference principle.

29. See, for instance, Barbara H. Fried’s “Rawls, Risk, and the Maximin Principle” in this recent collection.

30. This is indirectly acknowledged by Sunstein as well, though not via Pascal’s wager, in an otherwise sympathetic treatment of maximin (at least under certain limited circumstances, somewhat similar to those of Rawls and Gardiner) when he writes that if “technology offers the promise of “moonshots,” or “miracles,” offering a low probability or an uncertain probability of extraordinarily high payoffs,” as is certainly the case with AI, then “there are reasons to be cautious about imposing regulation.” In fact, if the moonshot gains are necessary in order to to avert plausible catastrophes, then the precautionary principle would compel us to not only refrain from regulating the technology but to actively promote its development. Note also that Sunstein does not address the de minimis difficulty.

31. With few exceptions, e.g., in classical physics the probabilities of certain outcomes can be computed exactly once the initial conditions are fixed (though, even there, measurement errors introduce some uncertainty in practice); in a tightly controlled manufacturing process it may be possible to compute the probabilities of defects with high precision given sufficient historical data and quality control measures; in genetics, the probabilities of inheriting certain traits can also be precisely calculated when the relevant genetic mechanisms are well understood; and so on.

32. See this article for a reply to Peterson, and this one for Peterson’s rejoinder.

33. The demand that the precautionary principle should not block moonshot benefits is markedly lenient when one considers the principle’s environmental origins, such as Talbot Page’s influential essay from 1978, where situations calling for precautionary intervention were thought to have only a “relatively modest benefit” associated with the regulatory gamble. This was due to an error asymmetry characterizing such situations: In matters of environmental policy, the price of a false negative is often severe, measured in lost human lives, whereas a false positive (such as banning a certain food additive) is only going to have economic costs, and probably minor ones at that. (From a machine learning perspective, we would say that the profit matrix is imbalanced.) Recall also Dr. Snow’s decision to remove the handle from the Broad Street water pump in Soho. The reasoning behind that intervention gamble was similar, due to the same asymmetry: If he was correct, he would stop the epidemic and avert a catastrophe, but if he was wrong he would only inconvenience the local residents for a while. The precautionary principle was not meant to apply when the cost of the intervention (whether an explicit cost or an opportunity cost) is tremendous, let alone boundless. At any rate, the role of the constraint here is simply to ensure coherence, so that the principle does not endorse two logically inconsistent courses of action.

34. For instance, it might not be necessary to commit to any one numeric threshold if it is agreed that the combined probability of the risk is high enough that it would exceed any reasonable value.

35. Of course, one could argue that some of these existential problems, such as global warming, could be solved without the help of ASI, but recall that the precautionary principle only needs a realistic possibility that they could not.

36. For the case of deterministic computation, refer to the notion of functional explanation discussed in Section 3 of this paper.

37. This scenario was recently echoed by Bengio.

38. Any intelligent organism would need trillions of cells.

39. In 2016, Hinton declared: “People should stop training radiologists now. It’s just completely obvious that, within five years, deep learning is going to do better.”

40. Alex Lewyt, president of vacuum cleaner company Lewyt Corp., in the New York Times in 1955.

41. Of course even that would not be enough on its own, as we have seen. There are additional conditions above the requirement that must be met for precautionary regulation to be rationally justified.