The lacklustre reception of the recent GPT-5 release¹ (Aug 2025) was a reminder of the slowing progress we can expect from future LLMs. The current LLM design has fundamental limits when it comes to achieving artificial general intelligence (AGI).

Besides not being able to learn easily after their very long and expensive training runs have completed, current LLMs seem unable to count² or do even simple arithmetic³. They fail at algorithmic reasoning tasks⁴ and give wrong responses with confidence (i.e. hallucinate). And they just can’t seem to follow the rules of chess⁵!

Therefore, most chatbot providers have now pivoted to agent-like systems that enrich the LLM’s context with past conversations, chain-of-thought, and tools. These tools are specifically designed to patch weaknesses of LLMs, such as math and code execution.

This sounds an awful lot like what I call the “Frankenstein stage of technology development”. There is little left to squeeze out of the core idea and the only progress consists of bolting on more stuff. Kind of like when mobile phones just started to get more cameras after the initial rapid improvements in terms of screen size, processing power and miniaturization had reached their limits⁶.

Agents. A Blast from the Past.

The idea of an “agent” was developed over 50 years ago to describe learning behaviour that involved repeated interaction with the environment and feedback or rewards. Both, psychology and early AI (or cybernetics) research adopted the concept and refined it⁷.

Richard S. Sutton formalized the concept of an autonomous agent in his 1983 PhD dissertation⁸: An agent interacts with an environment in discrete time steps $t$. The agent’s entire knowledge and perception of the environment (at time $t$) is captured in state $S_t$. Based on this state, the agent selects an action $a_t$. The agent then receives a state update $S_{t+1}$ and a reward $r_{t+1}$, which is key for reinforcement learning (RL).

Modern LLM-based chatbots (or so called “agentic” AI) draw on some but not all of the characteristics of this classic RL agent. They interact with the environment (i.e. the user) in discrete steps and their actions consist of generating tokens. Chatbots also update their state, e.g. by saving the user’s response to their conversational history. Additionally, they “act” by using tools and their “environment” can be their own output during chain-of-thought.

However, chatbots deviate from Sutton’s classic RL agent definition due to the lack of continuous online learning. Chatbots are mostly “fixed” at inference time. Reinforcement learning from human feedback (RLHF) is used during LLM fine-tuning, but not during the chat session.

The thumbs up/down that users can provide and other session metrics are probably anonymized and aggregated before being used for offline model fine-tuning. The closest chatbots come to learning within session is the way they change their behaviour based on the chat history and some user preferences (e.g. system prompt). But anyone who has tried to correct a chatbot knows the limits of the teaching-via-context approach. Without weight adjustments chatbots tend to eventually revert back to their original behaviour.

Conclusion

LLM progress has slowed even while the effort to improve the paradigm has increased tremendously. Currently billions of dollars are being invested in squeezing out marginal gains that don’t fundamentally make LLMs more intelligent⁹. This fact alone should give anyone pause who was hoping that LLMs could become AGI.

Of course, it’s possible that there will be another game-changer like the transformer architecture¹⁰, but this is increasingly unlikely given that thousands of publications each year haven’t delivered any major breakthrough.

The autoregressive transformer architecture¹¹ does not seem to be suitable for true intelligence, which is the ability to “achieve goals in a wide range of (novel) environments”¹². For now, LLMs mostly retrieve and interpolate between the language examples they were trained on. So by definition, they cannot generalize and give responses that are outside or in conflict with the training data.

While the LLM-chatbot loop sounds similar to the classic RL agent–environment interaction paradigm, it still differs fundamentally. Chatbots don’t learn online and even if they did it’s not clear if the classic RL agent approach would be feasible given the large number of states such a system could be in. I’m skeptical that LLMs themselves will be the key to AGI, despite the advances of LLM-based agents over “naked” LLMs.