‘We could lose control of future’: Sam Altman shares 2 ways AI could go rogue

‘AI must always serve people’

In a post on X, Altman said the first scenario is when humanity may lose control of the future to AI. To prevent that, he said, it is imperative for alignment and safety techniques to stay ahead of progress in model capabilities.

Also read: Sam Altman on awkward moment with Dario Amodei: 'I was confused'

“First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people. To ensure that, we need ways to ensure that alignment and safety techniques stay ahead of progress in model capabilities,” the post said.

Danger of power concentration

Altman said the second way in which AI progress may go rogue is when power, gained from the technology, gets concentrated in one place, creating a dystopian world.

“Second, we could end up in a world with too much concentration of power. If an extraordinarily powerful AI is used by one person or company to impress their worldview onto everyone else, the results could be extremely dystopian,” the post said.

Altman backs slowdown call

In a series of posts, Altman examined the challenges that futuristic versions of AI may pose while offering solutions to avoid them.

Cautioning that one lab or one country may end up getting too much power due to AI, Altman said avoiding these threats requires walking a narrow middle path.

Also read: Sam Altman: What's happening in India with AI is really amazing

In another post, he seconded Anthropic CEO Dario Amodei’s call to slow down the pace of AI development.

He said OpenAI, like Anthropic, will also employ third-party evaluators with employee-like access to audit adherence to safety measures, report on incidents and assess alignment of models during training.

“The world deserves confidence that American companies developing increasingly capable AI will act responsibly, especially as the trajectory of progress has steepened. Every frontier lab must deliver on this, and there is no reason any of us should come to work if we cannot,” he posted.

Also read: Sam Altman: ‘Listening to old people is biggest mistake’ for youth in AI era

Altman also called for a federal framework which could set consistent safety requirements for frontier AI. He, however, said companies needn’t wait for an anti-trust exemption or legislation to ensure safety, and should start managing frontier risks through consistent rules.

‘Pacing is not stopping’

Altman, in another post, however, clarified that slowing down AI progress doesn’t mean stopping it.

“When we talk about “pacing”, we do not mean “stopping”. Progress has been rapid and will continue to be. But it should be slower than it otherwise could be; interventions like safety cases and monitoring have significant costs,” he wrote.

Also read: Anthropic CEO warns AI could ‘take over internet’ without slowdown

“Pacing will be well worth this cost; no amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring,” he added.

Can AI kill? Scientists believe so

Altman’s caution came days after Anthropic CEO Amodei called for slowing the pace at which frontier AI capabilities advance to give safety measures time to catch up, proposing independent evaluators, coordination among leading AI companies and international cooperation.

Dropping a bombshell after his resignation a few days ago, Anthropic scientist Jacob Coxon had claimed people building AI believe that it could kill humanity by the end of the decade.

His claims were backed by Evan Hubinger, Alignment Science Lead of Anthropic.

Coxon, who has worked both in OpenAI and Anthropic had said that he doesn’t believe that any lab can safely build self-improving AI. He also accused the two companies of “racing straight to self-improving superintelligence” and jeopardising the future of humanity.