New MIT study confirms what you already know about AI: It doesn't understand anything

The latest AI models can produce output that is human-like and magical. But can they understand anything? According to the latest MIT study (via Techspot), it's a big fat no.

The key question is if the LLMs, or large language models, at the heart of the most powerful bots are able to construct accurate internal models of reality. The answer that MIT researchers came up with was largely no, they cannot.

The MIT team has developed new metrics to test AI that go beyond simple measures for accuracy in responses, and are based on what is known as deterministic Finite Automations, or DFAs.

A DFA is an interdependent problem that relies on a set rules. For the research, navigating the streets in New York City was selected as one of the tasks.

In New York City, the MIT team found that some generative AI models can provide very accurate driving directions under ideal conditions. Performance dropped when researchers added detours and closed some streets. The internal maps generated by LLMs during their training process were full of inconsistencies and non-existent streets.

"I was surprised at how quickly performance degraded once we added a diversion. If we close only 1 percent of possible streets, accuracy plummets immediately from nearly 100% to just 67.5%," says Keyon Vafa, lead author of the research paper.

The main lesson is that LLMs' remarkable accuracy in certain contexts may be misleading. "We often see these models doing impressive things and assume they understand something about the world. Ashesh Rambachan, senior author of the paper, says: "I hope we can convince people this is a very important question that we should think about carefully and not rely on intuitions.

This research serves as a reminder about what the latest LLMs are really doing. They are predicting the next word in a sequence using massive quantities of text that have been scraped, indexed and correlated. This process does not include reasoning or understanding.

This new MIT study showed that LLMs are capable of performing remarkably well even without understanding any rules. Despite this, accuracy can quickly deteriorate when faced with real-world variables.

This is not a surprise to anyone who has used chatbots. We've all seen how a seemingly coherent chatbot interaction can quickly degrade into gibberish or hallucinations after a certain type of interrogative questioning.

This MIT study helps to formalize that anecdotal knowledge. We all knew chatbots simply predict words. The incredible accuracy of some responses can sometimes convince you that there is something magical happening.

This latest study reminds us that it is almost certainly not. Not unless you're into mindless but incredibly accurate word prediction.