Large language models like ChatGPT have recently spooked a great many, and my Twitter feed is full of worriers saying how irresponsible orgs have been to make and release such models. Because, they say, such a system might have killed us all. And, as some researchers say that they are working on how to better control such things, worriers say we must regulate to slow/stop AI progress until such researchers achieve their goals. While I’ve written

Our best theories vary in generality. Some theories are very general, but most are more context specific. Putting all of our best theories together usually doesn’t let us make exact predictions on most variables of interest. We often express this fact formally in our models via “noise,” which represents other factors that we can’t yet predict.

Back in 2008 my ex-co-blogger Eliezer Yudkowsky and I discussed his “AI foom” concept, a discussion that we recently spun off into a book. I’ve heard for a while that Nick Bostrom was working on a book elaborating related ideas, and this week his

My ex-co-blogger Eliezer Yudkowsky last June: I worry about conversations that go into “But X is like Y, which does Z, so X should do reinterpreted-Z”. Usually, in my experience, that goes into what I call “reference class tennis” or “I’m taking my reference class and going home”. The trouble is that there’s an unlimited number of possible analogies and reference classes, and everyone has a different one. I was just browsing old LW posts today (to find a URL of a quick summary of why group-selection arguments don’t work in mammals) and ran across a quotation from Perry Metzger to the effect that so long as the laws of physics apply, there will always be evolution, hence nature red in tooth and claw will continue into the future – to him, the obvious analogy for the advent of AI was “nature red in tooth and claw”, and people who see things this way tend to want to cling to that analogy even if you delve into some basic evolutionary biology with math to show how much it isn’t like intelligent design. For Robin Hanson, the one true analogy is to the industrial revolution and farming revolutions, meaning that there will be lots of AIs in a highly competitive economic situation with standards of living tending toward the bare minimum, and this is so absolutely inevitable and consonant with The Way Things Should Be as to not be worth fighting at all. That’s his one true analogy and I’ve never been able to persuade him otherwise. For Kurzweil, the fact that many different things proceed at a Moore’s Law rate to the benefit of humanity means that all these things are destined to continue and converge into the future, also to the benefit of humanity. For him, “things that go by Moore’s Law” is his favorite reference class.

To extend our reach, we humans have built tools, machines, firms, and nations. And as these are powerful, we try to maintain control of them. But as efforts to control them usually depend on their details, we have usually waited to think about how to control them until we had concrete examples in front of us. In the year 1000, for example, there wasn’t much we could do to usefully think about how to control most things that have only appeared in the last two centuries, such as cars or international courts.

As a professor of economics in the GMU Center for the Study of Public Choice, I and my colleagues are well aware of the many long detailed disputes on the proper scope of regulation. One the one hand, the last few centuries has seen increasing demands for and expectations of government regulation. A wider range of things that might happen without regulation are seen as intolerable, and our increasing ability to manage large organizations and systems of surveillance is seen as making us increasingly capable of discerning relevant problems and managing regulatory solutions.

I’ve written much on my skepticism of local AI foom (= intelligence explosion). Recently I said that foom offers the main justification I understand for AI risk efforts now, as well as being the main choice of my Twitter followers in a survey. It was the main argument offered by

Paul Christiano was one of those who encouraged me to respond to non-foom AI risk concerns. Here I respond to two of his posts he directed me to. The first one says we should worry about the following scenario:

Years ago I was honored to share this blog with Eliezer Yudkowsky. One of his main topics then was AI Risk; he was one of the few people talking about it back then. We debated this topic here, and while we disagreed I felt we made progress in understanding each other and exploring the issues. I assigned a much lower probability than he to his key “foom” scenario.

Years ago my ex-co-blogger Eliezer Yudkowsky and I argued here on this blog about his AI risk fear than an AI so small dumb & weak that few had ever heard of it, might without warning, suddenly “foom”, i.e., innovate very fast, and take over the world after one weekend. I mostly argued that we have a huge literature on economic growth at odds with this. Historically, the vast majority of innovation has been small, incremental, and spread across many industries and locations. Yes, humans mostly displaced other pre-human species, as such species can’t share innovations well. But since then sharing and complementing of innovations has allowed most of the world to gain from even the biggest lumpiest innovations ever seen. Eliezer said a deep conceptual analysis allowed him to see that this time is different.

Years ago when the AI risk conversation was just starting, I was a relative skeptic, but I was part of the conversation. Since then, the conversation has become much larger, but I seem no longer part of it; it seems years since others in this convo engaged me on it.

Having hung around futurists and their audiences for four decades now I can tell you this: while most people are usually vaguely optimistic about the future, they change their minds fast when shown non-near futures described in plausible detail. (E.g.,

What I’m about to say is pretty obvious to most artificial intelligence (AI) researchers, but I think others might benefit from hearing it. In general, AI systems are created by combining computer hardware, other hardware, access to data, an architecture, parameter settings, and hand-made software. Machine learning (ML) systems have relatively little hand-made software, while robotic systems have more other hardware.

Compared to some, I am less worried about extreme near-term AI doom scenarios. But I also don’t like policy being sensitive to my or anyone else’s risk estimates. I instead prefer robust policies, ones we can expect to promote total welfare for a wide range of parameter values. We want policies that will give big benefits if foom risk is high, but impose low costs if foom risk is low. In that spirit, let me suggest as a compromise a particular apparently-robust policy for dealing with AI foom risk.

The appreciate an AI future, consider a future without them. That is, imagine that we somehow manage to forever prevent any AIs who might take control of civilization. But also assume that our descendants continue to evolve competitively, with technical and economic growth for millions or billions of years. Now ask: what exactly did you think would happen there?In my view, it is pretty obvious that our “human” descendants would change radically. Eventually for sure, but maybe even rather quickly. After all, we now differ greatly not only from our ancestors of ten million years ago, but even from our ancestors of a thousand years ago. Also, rates of change have been greatly accelerating, and new tech will let us make many deep changes to our bodies, brains, and environments. Our future civ should thus be far larger, more complex, and more varied. Faster rates of change and longer lifespans (likely including immortality) will let more different generations overlap in time, inducing more stronger conflicts between them. And in these conflicts, we expect later generations to be abler, richer, smarter, and nearer to the center of control and info. And thus to win. Some future changes will count as robustly-valuable innovations that accumulate and last a long time, much like the past innovations that we now embody and expect to long continue. Like eyes, consciousness, reason, laughter, love, law, markets, etc. But other changes may be less robust, varying more across space and time. Along those sorts of change dimensions, our descendants attitudes, habits, and values will “drift”, and without obvious limits. And some fiercely hate the prospect of such drift. To limit this drift, some hope that we will develop better “mind control” tech, together with strong civ-wide centralized governance. These would allow authorities to better control citizens, and a whole generation of parents to better control the next generation of kids. Such centralized powers seem to me both dangerous and distasteful. This mind control would at least partially displace past processes by which generations have chosen how to change their attitudes, styles, and values. Okay, now imagine a future date when becomes possible for some to live beyond Earth. Some may then reason:As an Earth person, I am partial toward Earth people, relative to space people. If we let people leave Earth, then space people will become different in some ways, grow more than Earth people, and eventually come to dominate. We should prevent this horror by preventing anyone from leaving Earth, at least until we find ways to centrally and absolutely mind-control all space people.This kind of partiality is infertile; genetic or memetic evolution does not in general select for it. Yes, this strategy might promote pro-Earth-people partiality, but it hurts all other partialities that would be helped by a much larger future economy. Now let’s consider AI. Humans now occupy a tiny corner of a vast mind space, and compared to our more human descendants, AIs would more expand out into this mind space, analogous to [...]

YouGov America released a survey of 20,810 American adults. … 46% say that they are “very concerned” or “somewhat concerned” about the possibility that AI will cause the end of the human race on Earth … There do not seem to be meaningful differences by region, gender, or political party. Younger people seem more concerned than older people.… Furthermore, 69% of Americans appear to support a six-month pause in “some kinds of AI development”. (