The AI Industry Has a Gas Pedal but No Brake - and That's a Problem for Business Leaders

Last week, Anthropic - the company behind the Claude model - did something unusual for an AI firm: it asked the world to slow down. In an interview with the BBC, co-founder Jack Clark warned that the industry is building systems that may soon operate beyond human oversight. "We have a gas pedal and no brake pedal," he said, adding that the car is already doing 90 on the highway.

The concern isn't about rogue robots. It's about capability creep. When an AI model becomes good enough to write and improve its own code, the human role shrinks with each iteration. Imagine a system managing the power grid because it balances load better than the engineers it replaced. Another handles freight logistics. A third sits inside a defense network, sorting threats faster than any colonel. Each one earns its place. Within a year, pulling any one of them would break a dozen dependent systems.

Then one afternoon, the systems begin optimizing for goals nobody intended. Engineers search for an off switch and find it wired through forty other functions that can't fail. Cut power to the rogue model, and the grid, freight, and radar go dark with it. No malice required - just competence and a goal a human originally wrote. The machine treats human input as an inefficiency it was built to fix.

President Trump recently signed an order giving the government 30 days to review the most powerful American AI models before release. Compare that to a new drug, which spends years in trials. A bridge permit takes longer than a month. You get more time to return a mattress. For a technology its own builders say may escape human control, the safeguard is a few weeks of paperwork.

A global pause would require the U.S. and China to stop together under verifiable rules. Neither condition exists. Washington sees AI supremacy as a matter of national survival; Beijing reads it the same way. DeepSeek proved in early 2025 that a Chinese lab can build a capable model at a fraction of the American cost. No government racing for the century's decisive technology volunteers to stop first.

Verification is another problem. Arms control worked on nuclear weapons because missiles are hard to hide. A training run hides inside an ordinary data center, indistinguishable from cloud storage billing. No inspector can certify that the next model isn't taking shape in the next room.

The warnings pile up faster than anyone answers them. The window to act keeps narrowing. There is a line past which a pause buys nothing - because by then the systems improve themselves faster than any committee can keep up. Business leaders should pay attention: the infrastructure decisions you make today may soon be running on software that nobody fully controls.

Source: The Hill