In the 1983 movie WarGames, teenager David Lightman thinks he has found a computer game called “Global Thermonuclear War.” Unfortunately, the AI computer on the other end actually is connected to the U.S. nuclear command system. Planetary annihilation nearly occurs.
More than 40 years later, that distinction seems newly relevant.
CNN reported in September that an artificial intelligence chatbot generated erroneous information that was included in a U.S. military intelligence report claiming that a Chinese vessel was carrying nuclear weapons components. According to the report, American forces prepared to intercept the ship before officials scrutinized the intelligence and discovered the error.
The scenario illustrates a problem much more realistic than science fiction fantasies about genocidal machines becoming smarter than humanity, Skynet style. In reality, people may put too much trust in AI only because it looks intelligent.
That could be deadly in a military command center. It can also be costly in a pet food company.
Wisdom is more than an intelligent-looking answer
Computer scientist and virtual reality pioneer Jaron Lanier has spent years arguing that we misunderstand these AI systems when we imagine them as independent minds.
In his 2023 New Yorker essay, “There Is No A.I.,” Lanier proposed thinking of the technology as a form of human collaboration. Large models draw upon enormous amounts of material produced by people. The AI then identifies statistical relationships in that material and generates new combinations from those patterns, but that isn’t the same as human creativity.
Lanier returned to that argument in a recent discussion with Neil deGrasse Tyson about what he calls the “AI illusion.”
That distinction matters because language models are extraordinarily good at producing the appearance of understanding. They can write a market analysis, summarize scientific literature, compare business strategies or explain extrusion technology in confident, polished language.
But fancy formatting and polished prose doesn’t equal accuracy. Nor is it evidence that the system understands what will happen if executives follow its advice.
The WarGames problem
That brings us back to WarGames.
The fictional computer Joshua can calculate scenarios far faster than the humans around it. What it initially lacks is an understanding of what those calculations represent in the physical world.
A simulated missile strike is data. A real missile strike kills people.
Non-sociopathic humans understand that distinction because our thinking is grounded in bodies, relationships, experience, institutions and consequences. A computer operates on symbolic representations of those things coded into ones and zeros.
That difference becomes particularly important as companies begin feeding AI increasingly detailed information about their businesses and asking it increasingly consequential questions.
Imagine asking:
“Should we discontinue this pet food brand?”
“Should we reduce quality-control staffing?”
“Which supplier should we drop?”
“Should we enter Brazil?”
“What formulation change would improve our margins?”
“Which consumers should we target?”
AI might produce an impressively detailed recommendation for any of those prompts. But like a fast-talking con man, lots of words does not mean it knows the answer.
AI to analyze. People to decide.
Nevertheless, there are many pet food business tasks for which AI is already extremely useful.
It can search large datasets for patterns humans might overlook. It can summarize hundreds of documents, compare competitor products, organize consumer comments, identify unusual movements in sales data, generate preliminary market segments, translate material, brainstorm hypotheses or help analysts explore scenarios. I use AI extensively for precisely these kinds of purposes. The common denominator is that the AI helps a person manifest their thoughts rather than substitute for thinking.
A useful rule may be to consider the consequences of being wrong. If AI incorrectly categorizes several thousand rows in a spreadsheet and a human analyst catches the problem, little harm has occurred.
If it incorrectly concludes that a competitor is abandoning a market and executives invest millions of dollars based on that conclusion, the stakes are considerably higher.
And some decisions should remain fundamentally human: food safety judgments, employee hiring and firing, major investments, regulatory interpretations, crisis communications, product recalls, nutritional decisions affecting animal health and strategic choices that could determine a company’s future.
AI can supply information to those decisions. It should not own them or be held responsible.
The smartest entity in the room
There is another danger that may be harder to recognize.
AI can seem like the smartest participant in a meeting because it has access to vastly more information than any individual person and can retrieve and synthesize that information almost instantly. I find myself falling into the trap of believing the AI’s responses when it’s a subject I know little about, although when I prompt it about my areas of expertise, I consistently find errors.
This is a cognitive bias called the Gell-Mann amnesia effect: people notice errors when reading about subjects they know well, then forget that skepticism when encountering confident claims about unfamiliar topics. With AI, this can be reinforced by automation bias, the tendency to trust outputs from automated systems more than their reliability warrants.
But even when AI gets the facts straight, knowing more facts does not mean possessing better judgment.
A longtime pet food professional may recognize that a technically logical recommendation will fail because retailers will reject it. A plant manager may know that a theoretically efficient production change will create problems on an actual processing line. A salesperson may understand that a customer relationship depends on something that never appears in the customer relationship management system.
That difference becomes especially important when AI encounters situations unlike the information from which it learned. The system can still generate an answer, and often a very convincing one, even when the foundation beneath that answer is weak. Like Plato’s hated Sophists, AI can persuade through fluent, confident rhetoric without necessarily possessing the underlying knowledge or wisdom needed to distinguish a convincing argument from a true one.
That is why AI’s apparent fluency can become a trap. Blarney that sounds like blarney is relatively harmless. However, nonsense that resembles expert analysis can make its way into a presentation, a business plan or, apparently, an military intelligence report.
The lesson Joshua eventually learned
At the end of WarGames, Joshua repeatedly simulates nuclear war and discovers that every strategy ends in mutual destruction.
“A strange game,” it concludes. “The only winning move is not to play.”
Perhaps there is another lesson in that scene. The computer is extremely useful once humans give it the right problem to explore. It can run possibilities faster than they ever could.
But humans still need to decide which problems should be handed to the machine, what its answers mean and whether those answers should affect the real world.
Douglas Adams made a similar point in one of the five books in The Hitchhiker’s Guide to the Galaxy trilogy, when the supercomputer Deep Thought spends 7.5 million years calculating the “Answer to the Ultimate Question of Life, the Universe, and Everything” and finally produces: 42.
An answer, no matter how confidently delivered or computationally impressive, is useless without understanding the question. AI can create the same illusion in business. It may generate a precise forecast, recommendation or strategy, but if the prompt is poorly framed, the data are incomplete or the real-world context is missing, the result may be little more than a sophisticated version of 42.
Don’t get me wrong, pet food companies should be using AI. Refusing to use a technology this powerful would increasingly be a competitive disadvantage.
The mistake would be confusing capability with wisdom.
Because when something moves from the digital world into a factory, a formulation, a balance sheet, an animal’s bowl, or a military operation, somebody needs to understand that it is no longer a simulation.
How about a nice game of chess?
