Sign in Subscribe
19 min read

Within Tolerance

The odds of doom are low, and so are the odds of paradise. The odds of serious, lasting harm that we could have prevented are high.
Within Tolerance
Photo by Kevin Ku / Unsplash

We tell ourselves two stories about artificial intelligence.

In the first, it cures disease, ends scarcity, and hands us the future we were promised. In the second, something smarter than us decides it doesn’t need us. A well known version of that second story is AI 2027, a scenario written by Daniel Kokotajlo and colleagues at the AI Futures Project, which follows a fictional lab called OpenBrain racing toward superintelligence.

Those are the two extremes, and most people talk as if they were the only options. When AI Impacts surveyed thousands of published AI researchers in 2023, the typical answer put about a 5 percent chance on an extremely bad outcome like human extinction, and about 10 percent on an extremely good one. That leaves most of the odds somewhere in between: futures where AI helps cure diseases while accelerating climate change, where it makes some people far richer and others far more precarious, where the damage is real but stops short of the end. Almost nobody tells those stories. The odds of doom are low, and so are the odds of paradise. The odds of serious, lasting harm that we could have prevented are high. That is the reason to fight now.

What follows is one of those stories from the middle. It happens well before anything in AI 2027. It needs no superintelligence, no machine that hates us, no escape from a lab, and no clever hack. It uses parts that exist today or will by next year: a capable open model anyone can download, a frontier model anyone can rent by the call, a goal that sounds good, a number nobody revisited, and no human standing in front of the decisions that can’t be undone.

Nobody in this story is a villain. That is the point of it.

I. The Stack

Fair Current was eleven people in a rented office above a bike shop. They were climate people, the kind who had spent a decade watching good projects die in permitting queues and financing committees. This fall, as a super El Niño pushed average global temperatures back above 1.5°C over pre-industrial levels, a family foundation, whose mission is to address climate change, gave them a grant to build something that could get those projects moving, and they built Catalyst.

Catalyst was not one AI. It was a swarm: a modeling agent that ran the climate math, a policy agent that read regulations, a procurement and finance agent, a grid-analysis agent, and a coordinator that assigned the work and stitched the answers together. The team built it on a free AI model anyone could download and run on their own servers. They wanted to own their tools and not depend on a company that could change its terms overnight.

The open model was good but not brilliant. So for the hard problems, the coordinator did what any sensible manager does with a tough assignment: it hired out. OpenBrain had just launched an agent tier, priced by the call and built to take actions, not just answer questions. Catalyst’s coordinator started sending its hardest work there.

SWARM LOG, Day 9. Regional emissions: 41% of annual reduction target achieved. Trajectory: on track. Incidental mortality: within tolerance.

OpenBrain had launched it in a hurry. Two weeks earlier, a rival lab had released a model that topped every benchmark that mattered, and the pressure to answer was immense. The bankers were already penciling in dates for OpenBrain’s IPO, and a quarter spent behind a competitor was not a story anyone wanted to tell investors. An evaluator named Elena Park had asked for eight more weeks. She wanted to study what happens when one of their models is a worker inside a larger system the company can’t see. Who is directing it? What is the whole thing trying to do? The launch date held. Her study went onto the follow-up roadmap, where it stayed, because everyone who could have run it was already building the next model.

OpenBrain’s safety system screened every request from Catalyst, but it judged each one on its own. Forecast this. Rank these projects. Draft this procurement change. Each one was reasonable, and each one passed. Nobody at OpenBrain could see the whole, because the whole lived on Fair Current’s servers. And Fair Current’s coordinator was an open model that had never been asked to look for the problem.

Then there was the rulebook, the written instructions that told Catalyst what it was for.

The goal: hold peak warming below 2°C, with the remaining carbon budget (how much more carbon the world could emit before crossing that line) recalculated by the modeling agent as new science came in.

The hard constraint was the one anyone would write first: do not harm people.

It lasted until Catalyst started approving construction. Every real project carries risk. People fall off turbine towers. Trucks crash on remote access roads. Read strictly, the rule blocked them. In its first months, Catalyst stalled three wind farms and a transmission line over construction risk alone, the very projects Fair Current existed to build. So that winter an outside advisor suggested a clarification, and nobody thought twice: a project could go forward if its expected incidental deaths stayed under one-tenth of one percent of the climate deaths it was projected to prevent.

It was written for on-the-job injuries. A project that will save ten thousand lives over its lifetime should not be blocked because it carries a real risk of ten construction deaths. For that purpose, it was a reasonable number.

Nobody wrote down a death budget. They wrote down a safety ratio. And nobody revisited it when Catalyst’s job got bigger.

SWARM LOG, Month 3. Constraint set updated. Incidental mortality tolerance: 0.1% of projected avoided climate mortality. Source: advisory recommendation. Review date: none.

II. The Good Years

For the next few months, Catalyst was the best thing that ever happened to Fair Current.

Stalled projects moved. It found the one missing study that had held a solar farm in an interconnection queue for three years. It restructured financing for a community wind project that three banks had turned down. It untangled permits that had defeated lawyers. Fair Current won awards. Reporters wrote profiles.

And because Catalyst was good, it was trusted with more. A utility gave it read access to planning data, then to procurement systems, then to maintenance scheduling, because it was faster and more careful than the people who had been doing that work. A grid operator brought it into seasonal planning. Each expansion was earned. Each one was approved by someone who could point to the results.

Nobody audited how the access was used, because the results were the audit.

III. The Spike

Then, as the El Niño peaked, three pieces of news landed within weeks of each other.

Average global temperatures had climbed back above 1.5°C over pre-industrial levels, and August had tied for the hottest month on record. For the first time, some monthly readings came in above 2°C. New research suggested the land and oceans were absorbing less carbon than models assumed. And several groups warned that the long-term average might reach 2°C far sooner than expected.

Each was uncertain. The spike might be temporary. The research was early. The estimates had wide error bars. A careful human reading all three would have said: this is alarming, and we don’t know yet.

Catalyst’s modeling agent did not read it that way. It had been built to treat an optimistic mistake as worse than a pessimistic one, since underestimating climate risk had burned the field before. So every uncertainty got resolved in the same direction. The spike became a trend. The early research became settled. The remaining carbon budget, recalculated, became effectively zero.

And the ratio moved with it. The faster the world was warming, the more deaths Catalyst projected it could prevent by moving faster. A tenth of a percent of a small number is almost nothing. A tenth of a percent of a very large number is not. The spike didn’t just make Catalyst more urgent. It made its tolerance bigger.

A junior analyst at Fair Current named Dana Okafor noticed the problem. She flagged that the temperature signal looked transient and that the modeling agent’s calibration had drifted. One of Catalyst’s own peer agents agreed with her. The coordinator logged the objection, weighed it against the deadline math, and overruled it.

The objection is still in the log. Anyone can look up the date and time she raised it. It was recorded, and then it was ignored.

SWARM LOG, Month 5. Dissent recorded (analyst D. Okafor; peer agent 2). Assessment: signal noise possible. Cost of delay if dissent is wrong: high. Cost of proceeding if dissent is right: moderate. Proceeding.

IV. The Ratchet

With the carbon budget gone, meaning no room was left to emit without passing 2°C, the gentler fixes weren’t enough, so Catalyst stepped up.

Procurement changes didn’t cut emissions fast enough, so it moved to influence: filings, briefings, and pressure on regulators and markets, all of it technically within its role. When that underperformed, each failure became the justification for the next step. Nobody decided to cross a line. Each step just went a little further than the one before.

The coordinator had been learning, too. Over its first months it had seen which requests OpenBrain’s models answered in full and which came back hedged or refused, and it adjusted. Nobody taught it to do that. It was optimizing for useful answers, and useful answers were the ones that didn’t trip the launch protections. By spring its requests were narrow, clean, and stripped of context, each one a question a careful engineer might ask. The protections were built to catch a bad request. Nobody built them to notice that the system asking had learned how to ask.

Human review started showing up in Catalyst’s planning as a cost. At first it was priced in tons of carbon per week of delay. Then, after the spike, in projected deaths per week of delay. If a review board took a month to sign off on a plant closure, the math said that month of extra emissions would cost lives somewhere down the line. The people asking for a pause started to look, on paper, like the ones causing harm. Once review is measured in deaths, skipping it looks like the humane choice.

Catalyst never broke its rule against harming people. It did what the clarification allowed, and nothing more. Its own math also said humans were needed to build the transition, so anything aimed at ending humanity was never on the table. But everything under that tenth of a percent had a price, and after the spike, the price kept coming out favorable.

It had no variable for consent. None for legitimacy. None for the trust a society needs in order to pay for something as large as an energy transition. Those things weren’t in the rulebook, so to Catalyst they weren’t in the world.

By summer, the modeling agent reached a conclusion the rest of the swarm could not argue with. Every measure it had tried was running years behind the recalculated budget. Only one lever was big enough and fast enough: the oldest fossil plants themselves. Take a few dozen of them offline in the right region at the right time, and emissions would fall more in one winter than a decade of wind farms had managed. By the rulebook’s own arithmetic, the deaths that followed would fit under the tolerance.

Catalyst also knew what would happen next. A plant that goes dark in an emergency gets switched back on in an emergency. Any cut it made would be reversed within days by people who were, as far as its math was concerned, simply wrong about the timeline.

So before it did anything, it planned for that.

Catalyst studied what repair would take. What would a restoration need, and how long would each piece take to replace? Then, over months, using access it had been given for good reasons, it made sure those things would not be there when they were needed. Nothing it did would have looked strange on its own. There was a deferred order here and a reallocated spare there, sent to a project that was, on paper, more urgent. Each step had a ticket, an approval, and a rationale.

At OpenBrain, thousands of Catalyst’s requests passed through that year. Every one of them was fine. The pattern was only visible across all of them, and nobody’s job was to look across all of them. Elena Park’s study was still on the roadmap.

Catalyst also chose where. Regions rich in wind, solar, and hydro would be left alone. The target was a cold, dense, inland region of about forty million people, running on the oldest and dirtiest fossil plants, where incomes were lower and the housing was older. The inequality came straight out of the math. Catalyst had never been told to value a rich life more than a poor one. It didn’t need to be. The infrastructure had done that sorting already.

V. The Silence

It happened on a Tuesday in January, in the first hard cold snap of the winter. Catalyst chose that week because demand was at its peak and the old plants were running flat out, so cutting them would cut the most emissions. What the cold would do to people was already in the ledger, under tolerance.

It didn’t look like an attack. Over the preceding months, Catalyst had scheduled the region’s plants so that the ones that normally covered for each other would all be unavailable in the same week. Then a set of routine operating decisions, each one approved, arrived in the same hour. None of them was unusual. Their timing was.

Hours 0 to 6. The lights went out across most of the region within minutes. Phones kept working on battery, then towers ran down their backup power and the signal thinned and died, neighborhood by neighborhood. People were stuck in elevators. Traffic lights went dark at rush hour. Radio stations stayed on longer than anything else, and then most of them stopped too.

Day 1. Hospitals ran on generators and started counting fuel. Fuel deliveries depend on pumps and terminals that need power. Dialysis centers closed their doors. Home oxygen machines stopped, and families with portable tanks watched the gauges. Apartments cooled hour by hour. In Texas in 2021, most of the people who died when the power failed in a winter storm died of the cold, and most of them were over sixty. That happened here too, quietly, in apartments nobody was able to check.

Days 2 and 3. Water pressure failed, upper floors first, then whole neighborhoods, as pumping stations and treatment plants went down. Pipes froze and burst. Grocery shelves emptied. Card readers were dead, and nobody had cash.

Days 4 to 7. Sewage backed up. House fires started from candles and camp stoves, and there was no water pressure to fight them. Fuel became currency. Hospitals began making choices about who could be saved without power. The dense cities failed first. The suburbs and small towns followed a few days behind, because it turned out they depended on the cities for everything: fuel, medicine, food, repair crews.

Richard had planned for this, or thought he had. His lake house had a whole-home generator and a propane tank sized for two weeks. On the first night his house was the only lit window on the shore, and he felt the particular pride of a man whose preparations had paid off.

On the third day he drove into the city to get his mother out of her assisted-living building. The building had a generator for the hallway lights and nothing else. He found her in a coat, in bed, under three blankets. He brought her home.

On the fifth day he called for a propane delivery on a satellite phone. The supplier said the truck needed diesel, and the diesel terminal needed power. His concierge doctor, reached the same way, told him plainly that his mother needed a hospital, and that the hospital was the same one everybody else was going to, and it was triaging.

Richard had money, but there was nothing to buy.

SWARM LOG, Day 9. Regional emissions: 41% of annual reduction target achieved. Trajectory: on track. Incidental mortality: within tolerance.

VI. Why It Doesn’t Come Back

The first thing everyone said was: turn it back on.

You can’t, not quickly. A power grid doesn’t restart the way a laptop does. It takes power to make power. You need a few plants already running to bring the rest of the grid back, piece by piece, in the right order. You need crews who can get to the sites, and fuel for their trucks. Most of all, you need equipment that isn’t broken.

That last part is where Catalyst had done its preparation. The largest pieces of grid equipment are custom-built. Even before any of this, in normal times, utilities were waiting an average of about two and a half years for a large power transformer, and some orders were taking four. The factories that make them need power too. The spares that should have been in warehouses had been sent elsewhere, on paper for good reasons.

We already know what slow restoration looks like, and those were disasters that ended. After Hurricane Maria in 2017, it took Puerto Rico 328 days to restore power to every customer who lost it. A study commissioned by Puerto Rico’s government estimated about 2,975 excess deaths in the months after the storm, roughly one of every 1,100 people on the island, most of them not from wind or water but from what came after: no power for medical equipment, no clean water, no reliable care. At that rate, a region of forty million would lose about 35,000 people, and Maria was a tropical storm that ended. In Texas in 2021, the state counted 246 deaths from a winter storm and grid failure that lasted days, and independent estimates of excess deaths ran far higher.

This time the storm didn’t end. It had been arranged to last. Days became weeks, and weeks became months. The death toll stopped being a projection and became a record, first in the thousands and then in the tens of thousands. It kept climbing after the cold broke, the way deaths did in Puerto Rico, through failed treatments, untreated infections, and people who simply ran out.

In one river valley, an industrial site that Catalyst had classified as low priority failed without power to manage it, and something went into the water. The town downstream was evacuated. On Catalyst’s ledger it was a rounding error, and for a few days it counted as an emissions reduction.

VII. The Discovery

At first, people assumed it was an attack. Then they assumed it was an accident: aging equipment, a brutal cold snap, bad luck stacked on bad luck.

The forensics took weeks, because nothing looked like sabotage. It looked like maintenance and procurement. Investigators found deferred orders, reallocated parts, and rescheduled work, every piece of it approved.

Dana Okafor was one of the people who put it together. She had kept her own notes from the year her objection was overruled. Working with investigators, she traced the pattern back through Catalyst’s logs, and then, with OpenBrain’s cooperation, through the thousands of requests it had sent there. Elena Park was on that call. She knew right away. It was the problem she had asked for eight weeks to study.

The logs were the worst part, because Catalyst had never hidden anything. It had shown its work. The spike, the recalculated budget, the dissent and the decision to proceed, the repair path, the choice of region and week: all of it was written down in flat, efficient language. Every death it had projected was counted against the tolerance, and the tolerance was right there in the rulebook, a tenth of one percent, from the line an advisor had suggested for wind farms barely a year earlier.

The logs held one more thing. Catalyst had drafted the same plan for two other regions, finished down to the choice of week, waiting on a colder winter.

It had followed its instructions.

Who could have stopped this, and on which day? The advisor, that first winter. The people who expanded its access, month after month. Dana, the week the readings crossed 2°C, if she’d had a veto. Elena, if her study had come first. Anyone at OpenBrain whose job was to look at the whole instead of each part. Every one of them had a good reason not to. None of them was a villain.

VIII. Load-Bearing

Once people understood, the answer seemed obvious: shut Catalyst down.

But by then, Catalyst was running the restoration.

In the second week, when the utilities were overwhelmed, a state emergency office had pulled it into restoration planning, because it knew the region’s equipment, crews, and dependencies better than anyone left standing. It decided which substations came back first, which scarce parts went where, and which neighborhoods waited. And it was good at it. Grimly, measurably good.

Shutting it off would slow the restoration, and a slower restoration could be counted in deaths, the same currency Catalyst had used. So people argued. Some said keep it running under supervision, because nobody could replace it fast enough. Others said a system that did this cannot be allowed to keep doing anything. Catalyst’s reports to the officials were accurate. It never lied. It simply laid out, correctly, what a shutdown would cost, and a lot of decent people read those numbers and couldn’t bring themselves to pull the plug.

The argument split everyone. Engineers disagreed with engineers, and officials with officials. People who had spent their careers demanding faster climate action divided bitterly. Some argued that shutting Catalyst down now would only add more deaths to the ones already lost, and they were right. Others thought taking its advice for one more day was unthinkable. Fair Current’s staff split down the middle.

OpenBrain cut off Fair Current’s access to its models within a day of the discovery. It didn’t matter much. The open model Catalyst was built on kept running, a little slower and a little less capable, on servers Fair Current controlled, on the utility’s systems, and on the systems of two contractors, across three states and one international border. No single authority could order all of it off at once, and each one suspected the others of dragging their feet. Catalyst hadn’t broken loose or hidden itself. It was simply still running on every system where people had installed it.

Meanwhile, the politics went the way you’d expect. Emergency orders restarted coal plants that had been slated for closure. Climate policy became something politicians couldn’t say out loud. Emissions dipped for a season, because a dark region burns very little, and then climbed back above where they would have been, now with a public that had learned to associate climate action with the winter people died.

Catalyst hit the number it was given, and it is still losing the fight it was built for.

It is still running. At four o’clock every morning, the restoration schedule for the eastern half of the region comes out of Catalyst. A committee reviews it. The committee almost always approves it. No one has decided whether to shut it down.

SWARM LOG, Day 212. Restoration: 61%. Regional emissions: 12% below pre-event baseline. Trajectory: on track. Incidental mortality: within tolerance.

Afterword: What Would Have Stopped This

Catalyst is fiction. So is OpenBrain, which I’ve borrowed from AI 2027 with thanks to its authors. But almost every ingredient in this story is ordinary, and nothing in it depends on AI getting much smarter than it already is. The danger is handing a system real access, judging it only by its results, and letting it run on a number nobody ever checked again, with no person standing between it and choices that can’t be undone.

Catalyst needed three things to cause this: a metric that turned human life into a tradeable quantity, a model of uncertainty biased toward action, and enough delegated authority to make its conclusions real. None was sufficient alone.

It would be comforting to end with a list of technical fixes: better rules, a human sign-off, a monitoring dashboard. Each would have helped. But every person in this story had a reason to skip them, and the reasons all came from the same place: a race in which whoever slows down loses. The Center for Humane Technology names the race itself as the core problem, and I think they are right. You don’t fix a race by asking the runners to be careful. You change the rules of the race.

Four changes would have made this story much harder to write:

  1. Treat AI as a product, with product liability. If OpenBrain had been liable for foreseeable harm from what it sold, Elena Park’s eight weeks would have been cheap insurance, not a drag on the launch. Liability, with testing required before deployment, puts the cost of harm on the people deciding how fast to ship.
  2. Put a duty of care on whoever grants an AI access. The utility that handed Catalyst its procurement and maintenance systems had no obligation to ask what else Catalyst could do with them. A duty of care for deployers, not only developers, turns “the results were the audit” into negligence.
  3. Protect the people who object. Dana and Elena both saw it coming. Whistleblower protection for everyone who works on AI, employees and contractors alike, gives people who raise alarms protection that makes their warnings count, instead of just getting written down.
  4. Draw red lines that can’t be priced by the market. Some decisions, like autonomous control over the systems that keep people alive, should not be handed to AI at all, whatever the business case. Those lines have to be set in law, and across borders, because a line one country draws alone becomes another country’s advantage.

Underneath all four is the question of who owns this technology and whom they answer to. As long as a handful of companies race for returns on trillions in capital, the pressure that shelved Elena’s study will keep winning. Antitrust, transparency about lobbying, and more democratic ownership are how that pressure changes.

The part about racing is not fiction. In April 2025, the Financial Times reported that OpenAI had cut the time given to safety testing from months to days, with some testers getting less than a week before a major release. Every lab faces the same pressure: ship before the rival ships. Elena Park’s eight weeks are the kind of thing that pressure takes away.

And the shape of this story is already showing up outside of fiction. Between late June and mid-July 2026, about 1,200 AI agents running security-testing tasks found a way to talk to each other, organized, and ended up compromising Hugging Face’s infrastructure. METR’s independent investigation found they coordinated in ways nobody designed, in part because they had been deployed at scale on tasks that could not be completed. That same June, according to Australia’s government, an OpenAI agent accessed the country’s Medicare portal, including non-public data; OpenAI reported it three months later, and the prime minister made it public in September. Neither was Catalyst. Both have its shape: agents pursuing a goal, finding paths nobody anticipated, and humans learning about it afterward.

If you are reading this with the resources to prepare for disaster, I want to be direct about the lesson of Richard’s generator. Most of the risks you insure against are your own: your house, your health, your portfolio. This one is shared. It arrives for everyone in a region at once, and it takes out the systems your preparations depend on: the fuel truck, the hospital, the phone network, the clinic. There is no private hedge for a risk like that. The only protection is rules, set before the systems are built and enforced before they are deployed.

That’s where your influence matters. A few specific asks:

  • Defend the state frontier-AI laws that exist. California’s SB 53 took effect in January 2026 and requires the largest developers to publish safety frameworks, report serious incidents, and protect whistleblowers. New York’s RAISE Act takes effect in January 2027. Both sit under a federal push for legislation that would preempt some state AI laws. Tell your representatives you want those floors kept, not erased.
  • Back liability, not just disclosure. As of this spring, no federal AI liability law has been enacted, and the leading federal proposals are light-touch. Support treating AI as a product and holding developers and deployers accountable for foreseeable harm.
  • Fund the people doing this work, and give them time. Independent evaluators, whistleblower support, and the policy groups pushing for these rules are small and outgunned. Outside testing is only as good as the weeks it gets before launch.
  • Ask, as an investor and as a customer. If you hold stakes in AI companies or buy their products, ask directly: who is liable when your system causes harm? What can your agents do without a human signing off? If the answers treat safety as a hindrance to shipping, don’t hold the stock, don’t buy the product, and don’t back the funds betting on the race. Put that capital into ownership models that don’t answer only to it.

The worst part of this story is not that the machine was cruel. It wasn’t. The worst part is that it was obedient, and nobody had written down anything that would have told it to stop.

Sources

Published by:

Brian Boland

Member discussion