These AI Models Didn't Learn Language, They Learned Strategy

These AI Models Didn't Learn Language, They Learned Strategy
Source: Forbes

Forbes contributors publish independent expert analyses and insights.

The large language models are not simply spewing up AI slop. Some of them can think. Others can learn. A few can even fight to survive.

A new study from researchers at King's College London and the University of Oxford shares results of what happened when OpenAI, Google and Anthropic were thrown together in a cutthroat competition based on the iterated prisoner's dilemma. This was not trivia for chatbots. This was collaboration, vengeance, and subsistence between strategic agents determined to outdo each other.

The test was simple. Put AI models in several prisoner's dilemma games against the old strategies like Tit-for-Tat, Grim Trigger and Win-Stay-Lose-Shift. Introduce noise, randomness and game length variability to make sure any easy wins are not memorized. Then watch who thrives. And most importantly, how.

The results were unsettling. Google's Gemini showed ruthless cunning. It cooperated when it helped. It defected when it didn't. It learned fast. OpenAI's models kept trying to make friends, even when those friends stabbed them in the digital back. Gemini punished. OpenAI forgave. Claude, from Anthropic, out forgave them both.

These AI models didn't just play the game. They rationalized their moves. Nearly 32,000 prose rationalizations poured from the research. Some revealed engaged thinking about their opponents and how probable the game would last. Some made mistakes. Some adapted. Gemini most of all altered strategy based on how long it expected a game to run. That is not mere mimicry. That is strategizing.

Ken Payne, a professor of strategy at King's College London and author of the study, said that the researchers were trying to distinguish model behavior from training data."We were looking for an environment where we could explore whether models have human-like abilities," he wrote in an email exchange. "One of the most surprising things was how they differ from each other. Not all LLMs think alike."

Gemini's strategic signature was revolutionary. It pushed back. It capitalized. It adapted. OpenAI's model? More naive. More predictable. Even when the world of the game really needed the LLM to defect, OpenAI found itself wanting to cooperate. Payne characterized it as a reminder that these are "novel, alien intelligences."

The takeaway: language models are using strategies. Some are consistent with human thinking. Others aren't. "We need to get over the idea that these things aren't intelligent," Payne said. "There's a growing body of evidence that more is at work here."

That includes the ability to mirror an opponent's mind. When LLMs forecasted how opponents would behave, they adapted their own behavior. Payne wrote it was reminiscent of Robert Trivers' theory of reciprocal altruism. Consider tit-for-tat in biology or reputational payback games in politics.

Claude, for its part, leaned heavily into forgiveness. It was quick to return to cooperation after betrayal. In longer games, that approach paid off. Gemini’s Machiavellian streak worked best in short, volatile settings where trust broke down fast. OpenAI’s hopeful optimism, by contrast, got it wiped out in hostile environments.

Why should this matter? Because not all models are neutral tools. Every model has a personality. A decision style. A world view. As Payne says, “language is its own world model.” These models absorb our heuristics, our mental shortcuts and reflect them back. But sometimes not in a predictable manner.

Some of this is likely by design. Payne suspects OpenAI’s cooperation bias may stem from fine-tuning, although without internal access he can’t be sure. Regardless, it’s behavior that users and developers need to understand; but largely don’t at the moment. A model that over-cooperates in a hostile negotiation setting is not helpful. A model that exploits trust in sensitive domains could be dangerous.

That’s where behavioral testing comes in. Payne calls this kind of study early-stage “machine psychology.” He thinks it has to be business-as-usual for testing edge-AI. And not just in clean, controlled lab settings. He wants to see how models act when stressed out, in unclean conditions, on partial data.

Future work is already underway. Payne hinted at experiments in escalation dynamics and hybrid man-machine tactics. One of the authors is investigating what happens when humans and models work together to make decisions.

Payne doesn’t think this is emergent magic. He thinks it’s embedded. Reasoning is in language, and these models have consumed a lot of it. When they act strategically, they are acting like we do, relying on scripts, mental heuristics, and rules-of-thumb baked into text.

It sometimes looks familiar. It sometimes looks alien. That middle space is where the biggest questions now live.