Exercise tips, dinner suggestions, or help with school or work. Large language models (LLMs) like ChatGPT, DeepSeek and Gemini are designed to help us with many different tasks and problems.
But how good are they at solving logical problems? And what affects their reasoning capabilities?
This is what researcher Daniel Kaiser explores in a recently published study. He is researching LLMs as part of his doctoral project at UiT's Machine Learning Group and Integreat – Norwegian Centre for Knowledge-Driven Machine Learning.
In the study, he has developed a method to examine the logical problem-solving and reasoning skills of LLMs. His research is published as a paper for the prestigious International Conference on Learning Representations (ICLR) 2026.
Uncovers hidden limitations in LLMs
While LLMs have become a useful technology with several benefits, they are known to make mistakes. Sometimes in a catastrophic manner.
Earlier in 2025, ChatGPT made up 11 out of 18 sources which Tromsø municipality used for a new school structure report – a typical example of a hallucination.
"One should never blindly trust what an LLM says – even if it appears true or convincing. It's important to always check and verify their answers," Kaiser warns us.
He believes the method, named CogniLoad, can help detect and understand the limitations models face when solving logical problems.
"It is made to help us understand why certain LLMs excel or fall short on different tasks," he adds.
Not all LLMs are good at the same task
An LLM's design, such as its model size and training data, determines its ability to help us with a particular problem. In other words, not every LLM is equally suited for the same task.
"There are huge differences in what different LLMs are capable of. Big models like ChatGPT's GPT-5 model tend to excel at advanced and difficult problems, while smaller models like Meta's LLaMA models are more suited for easier ones," Kaiser explains.
But it isn't always obvious what kind of tasks certain LLMs are best or worst at. The models' complex structure also makes it difficult to understand where potential mistakes come from.
Valuable knowledge
Therefore, it's important to find out what different LLMs can and cannot do, regardless of how advanced they are. Even the largest models can still make mistakes – despite their confident tone.
"A test like CogniLoad can help pinpoint where a model's reasoning breaks down. This makes it possible to examine what kinds of logical mistakes LLMs make," Kaiser says.
This knowledge is valuable in many different ways.
"We can use it to understand the LLMs' limitations and what they struggle the most with. Developers can use this information to adjust their models to make them better," he adds.
Logical riddle
CogniLoad involves giving LLMs a logical riddle. It starts by describing a situation with several people and facts about them – like what they are wearing or what music they last listened to.
Then the model is given a series of statements which repeatedly changes what this situation looks like. At the end, the chatbot is asked one specific question about a person, like what color their socks have.
"To get it right, the chatbot has to keep track of all these changes from start to finish without making any mistakes," Kaiser explains.
The researchers can adjust the riddle to make it more challenging, such as by increasing its length or complexity, or by adding more irrelevant information. This tunability is designed to reveal what aspects of the riddle affect the LLM's ability to solve it.
The method is based on a theory on human cognition called Cognitive Load Theory; which states that how hard our brain has to work affects our ability to solve different tasks.
"When we have too much to keep in mind at once, it becomes harder to reason carefully and avoid mistakes. Since AI systems like LLMs are designed to imitate human intelligence, we wanted to look at how different types of cognitive load affect an LLM's reasoning abilities," he adds.
Tested on ChatGPT, DeepSeek and Gemini
Kaiser tested the method on 22 different LLMs – both on open and commercial models like ChatGPT, DeepSeek, and Gemini.
"The point was to see what kinds of pressure these different models handle well, and what kinds make them struggle," Kaiser explains.
Findings show that the method can provide unique insight into how these LLMs process and solve logical problems – regardless of their size.
"They show that we can apply this method on all these different models to understand which aspects of the problem affect their ability to reason," says Kaiser.
.jpg)
Similarities to human intelligence
The results reveal some interesting similarities between how humans and LLMs process information.
"We see that factors such as length, complexity and noise do in fact affect the LLMs' ability to solve logical problems. Just like when humans are exposed to different forms of cognitive workload," Kaiser says.
Even the largest LLMs struggled when the task was made longer or more difficult.
"It's a reminder that even when the best chatbots sound confident and fluent, they can still lose track of details and end up wrong," Kaiser says.
Model size is important
Adjusting the riddle's length caused the most issues for the LLMs. However, the size of the models does matter.
"The longer the puzzle got, the harder it became for many models to give an accurate answer. We see that smaller models tended to struggle sooner, while larger models could follow the chain for longer," he says.
"But eventually, the best models started making more mistakes when the task became quite lengthy," Kaiser adds.
He observes a similar pattern when tuning the riddle's complexity.
"Accuracy also drops off when the statements became harder to follow and required more careful tracking," Kaiser says.
CogniLoad is not meant to measure what the LLM already knows, Kaiser explains – but to study how well LLMs reason when encountering new information.
"It is not a test of knowledge where we quiz the LLMs about facts they are supposed to remember. In this case, we look specifically at how well the models do when facing a problem they've never seen before," he says.
Is Artificial General Intelligence closer than we think?
AI systems develop rapidly, and some people fear they will soon match or surpass human intelligence – achieving so-called Artificial General Intelligence (AGI).
While CogniLoad doesn't provide a clear answer about the future, Kaiser's research still suggest that this imagined scenario is far beyond the horizon.
"Even puzzles that sound simple can become difficult for today's models when you make them longer and harder to follow. The riddle should actually be pretty simple for an LLM to solve, so it's quite fascinating to see that even the big and expensive LLMs found it challenging when we increased the difficulty," Kaiser says.
Both small and large LLMs still have plenty of room for improvement, he adds.
"It shows in a way how far away even today's AI models are from achieving AGI," Kaiser laughs.
Reference:
Daniel Kaiser et. al: CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density. The Fourteenth International Conference on Learning Representations (ICLR 2026)