Chain Of Thought For Reasoning Models Might Not Work Out Long-Term

Forbes

01-07-2025

chain, isolated, 3d rendering
New reasoning models have something interesting and compelling called 'chain of thought.' What that means, in a nutshell, is that the engine spits out a line of text attempting to tell the user what the LLM is 'thinking about' as it completes a task.
For instance, if you ask a model a question like: 'what does (X) person do at (X) company?' you might get a chain of thought with items like this, given the system knows how to find the relevant info:
That's a simple example, but chain of thought has been something people rely on quite a bit over the last couple of years.
However, experts are now looking at the limitations of chain of thought reasoning, and suggesting that maybe this resource is lulling us into a false sense of security when it comes to trusting the results that we get from AI.
Language is Limited
One way to describe the limitations of reasoning chains of thought is that language itself is not precise, or particularly easily benchmarked.
Language is clunky. And there are hundreds of languages used across the globe. So the idea that a machine could precisely explain its working in a particular language is an idea with various strict limitations.
Take a look at this excerpt from a paper released by Anthropic, which is actually an academic treatise written by a number of authors.
This and other sources suggest that the chain of thought is just not sophisticated enough to really be accurate, especially as models get larger and demonstrate higher performance levels.
Or check out this idea posed by Melanie Mitchell at Substack in 2023, as these CoT systems were really about to take off:
'Reasoning is a central aspect of human intelligence, and robust domain-independent reasoning abilities have long been a key goal for AI systems,' Mitchell wrote. 'While large language models (LLMs) are not explicitly trained to reason, they have exhibited 'emergent' behaviors that sometimes look like reasoning. But are these behaviors actually driven by true abstract reasoning abilities, or by some other less robust and generalizable mechanism—for example, by memorizing their training data and later matching patterns in a given problem to those found in training data?'
Mitchell then asked why this matters.
'If robust general-purpose reasoning abilities have emerged in LLMs, this bolsters the claim that such systems are an important step on the way to trustworthy general intelligence,' she added. 'On the other hand, if LLMs rely primarily on memorization and pattern-matching rather than true reasoning, then they will not be generalizable—we can't trust them to perform well on 'out of distribution' tasks, those that are not sufficiently similar to tasks they've seen in the training data.'
Testing Faithfulness?
Alan Turing came up with the Turing test in the mid-1900s – the idea that you can measure how good computers are at acting like humans. There are also a lot of things that you can also measure about LLMs using high-level test sets – you can tell how good the models are at math, or at higher-level thought problems.
But how do you tell if the machine is being truthful, or in the words of the authors, 'faithful'?
The above paper goes into the ins and outs of testing faithfulness of LLM models. When I read this entire explanation, what I got is that faithfulness is subjective in a way that math and stochastics are not. Then we have a limited ability to really understand whether machines are being faithful to us in their results.
Think of it this way – we know that for their responses to our questions or statements, LLMs are going on the Internet and looking very broadly at what humans have written. Then they're imitating that. So they're imitating the technically correct knowledge, they imitate the way in which they produce results, they imitate the ways that humans talk – And they also might imitate the ways that humans hedge, the ways that humans hide information, the sins of omission for which we always castigate the media, and a human's propensity to just lie or dissemble in quite sophisticated, or alternately, simple ways.
Chasing Incentives
Furthermore, the authors of the paper point out that LLMs might chase incentives in the same way humans do. They might highlight some particular information that's inaccurate, or less accurate, to get a reward or a prize. Again, they may have learned this directly from us, from how we act on the Internet.
The authors of the above paper call it 'reward hacking.'
'Reward hacking is an undesired behavior:' they write. 'Even though it might produce rewards on one given task, the behavior that generates them is very unlikely to generalize to other tasks (to use the same example, other video games probably don't have that same bug). This makes the model at best useless and at worst potentially dangerous, since maximizing rewards in real-world tasks might mean ignoring important safety considerations (consider a self-driving car that maximizes its 'efficiency' reward by speeding or running red lights).'
At best, useless, and at worst, potentially dangerous. That does not sound good.
Philosophy of Technology
Here's another important aspect of this that I think deserves attention.
This whole idea of evaluating chains of thought is not a technical idea. It doesn't have to do with how many parameters the machine has, or how they're weighted, or how to do particular math problems. It has more to do with the training data and how it's used intuitively. In other words, this debate covers more of the black box of stuff that quants don't engage in when they evaluate models.
That makes me think that we actually do need something I've called for many times – a new army of paid philosophers to figure out how to interact with AI. Rather than mathematicians, we need more people who are willing to think deeply and apply human concepts and ideas, often intuitive ones, and ones based on society and history of civilization, to AI. We are woefully behind in this, because we've solely focused on hiring people who can write Python.
I'll get off the soapbox, but in figuring out how to move beyond chains of thought, we may end up having to realign more of our efforts when it comes to AI job training.