Today's AI development methods risk training AIs to be deceptive, manipulative and ambitious. This might not be easy to fix as it comes up.

Hypothetical stories where the world tries, but fails, to avert a global disaster.

Crossposted from the AI Alignment Forum . May contain more technical jargon than usual. In late 2022, Nate Soares gave some feedback on my Cold Takes series on AI risk (shared as drafts at that point), stating that I hadn't discussed what he sees as one of the key difficulties of AI alignment.  I wanted to understand the difficulty he was pointing to, so the two of us had an extended Slack exchange, and I then wrote up a summary of the exchange that we iterated on until we were both reasonably happy with its characterization of the difficulty and our disagreement. 1 My short summary is:  Nate thinks there are deep reasons that training an AI to do needle-moving scientific research (including alignment) would be dangerous. The overwhelmingly likely result of such a training attempt (by default, i.e., in the absence of specific countermeasures that there are currently few ideas for) would be the AI taking on a dangerous degree of convergent instrumental subgoals while not internalizing important safety/corrigibility properties enough.  I think this is possible, but much less likely than Nate thinks under at least some imaginable training processes.

Push AI forward too fast, and catastrophe could occur. Too slow, and someone else less cautious could do it. Is there a safe course? Source: Crossposted from the Cold Takes Audio podcast. --- Share feedback on this narration .

In previous pieces, I argued that there’s a real and large risk of AI systems’ developing dangerous goals of their own and defeating all of humanity - at least in the absence of specific efforts to prevent this from happening. A young, growing field of AI safety research tries to reduce this risk, by finding ways to ensure that AI systems behave as intended (rather than forming ambitious aims of their own and deceiving and manipulating humans as needed to accomplish them). Maybe we’ll succeed in reducing the risk, and maybe we won’t. Unfortunately, I think it could be hard to know either way . This piece is about four fairly distinct-seeming reasons that this could be the case - and that AI safety could be an unusually difficult sort of science. This piece is aimed at a broad audience, because I think it’s important for the challenges here to be broadly understood. I expect powerful, dangerous AI systems to have a lot of benefits (commercial, military, etc.), and to potentially appear safer than they are - so I think it will be hard to be as cautious about AI as we should be. I think our odds look better if many people understand, at a high level, some of the challenges in knowing whether AI systems are as safe as they appear. Source:

An overview of key potential factors (not just alignment risk) for whether things go well or poorly with transformative AI.

This post is part of my AI strategy nearcasting series: trying to answer key strategic questions about transformative AI, under the assumption that key events will happen very soon, and/or in a world that is otherwise very similar to today’s. Cross-posted from Less Wrong and Alignment Forum. This post gives my understanding of what the set of available strategies for aligning transformative AI would be if it were developed very soon, and why they might or might not work. It is heavily based on conversations with Paul Christiano, Ajeya Cotra and Carl Shulman, and its background assumptions correspond to the arguments Ajeya makes in this piece (abbreviated as “Takeover Analysis”). I premise this piece on a nearcast in which a major AI company (“Magma,” following Ajeya’s terminology) has good reason to think that it can develop transformative AI very soon (within a year), using what Ajeya calls [...]

I sometimes hear people asking: “What is the plan for avoiding a catastrophe from misaligned AI?” This post gives my working answer to that question - sort of. Rather than a plan, I tend to think of a playbook.1 A plan connotes something like: “By default, we ~definitely fail. To succeed, we need to hit multiple non-default goals.” If you want to start a company, you need a plan: doing nothing will definitely not result in starting a company, and there are multiple identifiable things you need to do to pull it off. I don’t think that’s the situation with AI risk. As I argued before, I think we have a nontrivial chance of avoiding AI takeover even in a “minimal-dignity” future - say, assuming essentially no growth from here in the size or influence of the communities and research fields focused specifically on existential risk from misaligned AI, and no highly surprising research or other insights from these communities/fields either. (This statement is not meant to make anyone relax! A nontrivial chance of survival is obviously not good enough.) I think there are a number of things we can do that further improve the odds. My favorite interventions are such that some success with them helps a little, and a lot of success helps a lot, and they can help even if other interventions are badly neglected. I’ll list and discuss these interventions below. So instead of a “plan” I tend to think about a “playbook”: a set of plays, each of which might be useful. We can try a bunch of them and do more of what’s working. I have takes on which interventions most need more attention on the margin, but think that for most people, personal fit is a reasonable way to prioritize between the interventions I’m listing. Below I’ll briefly recap my overall picture of what success might look like (with links to other things I’ve written on this), then discuss four key categories of interventions: alignment research, standards and monitoring, successful-but-careful AI projects, and information security. For each, I’ll lay out: How a small improvement from the status quo could nontrivially improve our odds. How a big enough success at the intervention could put us in a very good position, even if the other three interventions are going poorly. Common concerns/reservations about the intervention. Overall, I feel like there is a pretty solid playbook of helpful interventions - any and all of which can improve our odds of success - and that working on those interventions is about as much of a “plan” as we need for the time being. The content in this post isn’t novel, but I don’t think it’s already-consensus: two of the four interventions (standards and monitoring; information security) seem to get little emphasis from existential risk reduction communities today, and one (successful-but-careful AI projects) is highly controversial and seems often (by this audience) assumed to be net negative. Many people think most of the above interventions are doomed, [...] Source: Did you find this narration helpful? How could it be improved? As an experiment, we’re releasing AI narrations of all new posts with >125 karma on the “EA Forum (All audio)” podcast. Please share your thoughts and/or report bugs in the narration . Narrated by TYPE III AUDIO .

A few ways we might get very powerful AI systems to be safe.

This post summarizes the way I currently think about career choice for longtermists. I have put much less time into thinking about this than 80,000 Hours, but I think it’s valuable for there to be multiple perspectives on this topic out there. Edited to add: see below for why I chose to focus on longtermism in this post. While the jobs I list overlap heavily with the jobs 80,000 Hours lists, I organize them and conceptualize them differently. 80,000 Hours tends to emphasize “paths” to particular roles working on particular causes; by contrast, I emphasize “aptitudes” one can build in a wide variety of roles and causes (including non-effective-altruist organizations) and then apply to a wide variety of longtermist-relevant jobs (often with options working on more than one cause). Example aptitudes include: “helping organizations achieve their objectives via good business practices,” “evaluating claims against each other,” “communicating already-existing ideas to not-yet-sold audiences,” etc. (Other frameworks for career choice include starting with causes (AI safety, biorisk, etc.) or heuristics (“Do work you can be great at,” “Do work that builds your career capital and gives you more options.”) I tend to feel people should consider multiple frameworks when making career choices, since any one framework can contain useful insight, but risks being too dogmatic and specific for individual cases.) For each aptitude I list, I include ideas for how to explore the aptitude and tell whether one is on track. Something I like about an aptitude-based framework is that it is often relatively straightforward to get a sense of one’s promise for, and progress on, a given “aptitude” if one chooses to do so. This contrasts with cause-based and path-based approaches, where there’s a lot of happenstance in whether there is a job available in a given cause or on a given path, making it hard for many people to get a clear sense of their fit for their first-choice cause/path and making it hard to know what to do next. This framework won’t make it easier for people to get the jobs they want, but it might make it easier for them to start learning about what sort of work is and isn’t likely to be a fit. I’ve tried to list aptitudes that seem to have relatively high potential for contributing directly to longtermist goals. I’m sure there are aptitudes I should have included and didn’t, including aptitudes that don’t seem particularly promising from a longtermist perspective now but could become more so in the future. In many cases, developing a listed aptitude is no guarantee of being able to get a job directly focused on top longtermist goals. Longtermism is a fairly young lens on the world, and there are (at least today) a relatively small number of jobs fitting that description. However, I also believe that even if one never gets such a job, there are a lot of opportunities to contribute to top longtermist goals, using whatever job and aptitudes one does have. To flesh out [...]

The long view of economic history says we’re in the midst of a huge, unsustainable acceleration. What happens next? Source: Crossposted from the Cold Takes Audio podcast.

PASTA: Process for Automating Scientific and Technological Advancement. Source: Crossposted from the Cold Takes Audio podcast.

What the best available forecasting methods say about when we can expect transformative AI - and why there’s no “expert field” for this topic. Source: Crossposted from the Cold Takes Audio podcast. --- Share feedback on this narration .

We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. Source: Crossposted from the Cold Takes Audio podcast. --- Share feedback on this narration .