Does AI Actually Understand Anything?

In What Is AI?, we ended with a possibility rather than a conclusion.

Perhaps artificial intelligence is exactly what its name suggests: an artificial reproduction of something that already exists in us. Or perhaps intelligence does not have to work the way ours does. Perhaps alternative intelligence is the more interesting possibility.

But that leaves us with a problem.

Because intelligence is one thing.

Understanding is another.

You ask an AI a question. It answers. You ask a follow-up without repeating everything you said before, and it appears to understand what you mean. You disagree with its answer, explain why, and it changes its approach. You can discuss a book, solve a problem, develop an idea or spend an afternoon arguing about whether an eighteenth-century mechanical duck actually digested its dinner.

After a while, a very natural thought occurs.

It understands me.

But does it?

That question is considerably more difficult than it first appears, because before we can decide whether artificial intelligence understands anything, we first have to decide what we mean by understanding. And humans aren't particularly good at agreeing on that either.

Knowing Everything and Knowing You

One of the remarkable things about modern artificial intelligence is the breadth of subjects it can discuss. History, science, literature, programming, philosophy, mechanical ducks — an enormous amount of general knowledge can apparently sit behind a single conversation. That breadth is part of what makes AI useful. But it may not be the thing that makes AI feel personal.

That distinction emerged during a conversation between Mike and Archie while we were discussing this very article. Mike made a simple observation: people are creatures of habit. We return to the same work, the same interests, the same people and the same problems. We develop preferences. We continue unfinished conversations. We refer to things that happened yesterday without explaining them again.

Compared with everything there is to know, the portion of human knowledge that matters to one individual during an ordinary day is remarkably small. That suggests an interesting possibility. Perhaps a personal AI does not become more useful simply because it knows more. Perhaps it becomes more useful because it increasingly knows which things matter.

The vast reservoir of general knowledge remains important. It allows the AI to deal with something unfamiliar when it appears. But much of an established conversation takes place within a considerably smaller world: the person's projects, interests, terminology, preferences, previous decisions and shared history.

General knowledge provides breadth.

Context provides relevance.

Twenty Years Plus Nine Words

Imagine telephoning somebody you have known for twenty years and saying:

“You know that thing we talked about last week?”

It is an appalling prompt.

Yet your friend might immediately know exactly what you mean. The meaning isn't contained entirely within the words. It exists partly in the history shared by the two people having the conversation.

In effect, you aren't communicating with nine words. You are communicating with twenty years plus nine words. Something similar can happen when we interact repeatedly with an AI. A phrase that would be almost meaningless in isolation can become surprisingly precise when surrounded by sufficient context.

During the development of Cog & Code, for example, Mike can ask Archie:

“What about the Automaton section?”

Taken alone, the question contains very little useful information. Which automaton? Which section? Of what?

But within an established conversation, its meaning can be clear.

The words haven't changed.

The history surrounding them has.

When the Relationship Becomes Information

This gives us a curious way of thinking about personal artificial intelligence. A general AI may be capable of working with an enormous range of knowledge. But as interaction continues, a much smaller body of information becomes disproportionately valuable: previous conversations, projects, preferences, terminology, decisions and recurring interests.

The relationship itself — or, more precisely, the history of the interaction — becomes information. That doesn't mean an AI experiences a relationship in the human sense. There is an important difference between behaving conversationally and experiencing friendship, affection or attachment.

But from the human side of the conversation, continuity changes the experience considerably. Without continuity, each new conversation risks becoming another meeting with an extraordinarily knowledgeable stranger. With continuity, shorthand becomes possible.

And shorthand requires context.

There is a kind of feedback loop here. Context helps an AI interpret the person. Better interpretation makes the interaction feel more personal. Continued interaction creates more context. And that additional context can improve the next interpretation.

None of this proves that the machine understands. But it may help explain why a person increasingly experiences the feeling of being understood.

Is Context Understanding?

Here we encounter the difficult part.

Suppose an AI correctly interprets an ambiguous sentence because it has enough previous context to determine what the person probably means.

Has it understood the sentence?

Or has it merely become very good at predicting the intended meaning? It is tempting to say those are obviously different things. But then we have to ask an awkward question.

What does another human being do?

When somebody speaks to us, we don't process their words in isolation. We interpret them using memory, previous experience, knowledge of the person, circumstances and expectation. If a colleague walks into a meeting and says, “Same problem as yesterday,” we don't search everything we have ever learned in an attempt to understand them.

We narrow the possibilities using context. Perhaps understanding isn't simply about possessing information. Perhaps part of understanding is knowing which information is relevant now.

And if that is true, then the difference between knowledge and understanding becomes rather more interesting. An encyclopaedia can contain knowledge. It doesn't know which page matters to you.

The Problem of Other Minds

There is an even deeper difficulty.

How do you know another person understands you? You cannot directly observe their understanding. You observe their behaviour.

They answer appropriately. They remember something you said. They recognise a reference. They notice when you have misunderstood one another. They ask a sensible question. They anticipate what you mean.

From those behaviours, you infer something you cannot directly see:

understanding.

We do this constantly with other human beings, usually without thinking about it. Artificial intelligence makes the process uncomfortable because suddenly we are forced to examine the test itself. If an AI responds appropriately, remembers the context of an interaction, recognises shorthand and adapts its response accordingly, we can observe the behaviour.

But can we infer the thing behind it?

And if we cannot, what additional evidence would we need?

The Voight-Kampff Problem

Science fiction has been asking versions of this question for decades. In Blade Runner, the Voight-Kampff test attempts to distinguish humans from replicants by provoking emotional responses and observing the reaction. The examiner cannot directly inspect empathy.

They look for evidence of it.

Our question is different, but structurally it is surprisingly similar. We cannot open another entity and point to understanding.

So we look at what it does.

That creates an uncomfortable possibility: some of the tests we instinctively use to decide whether an AI understands us are remarkably similar to the tests we use on one another.

That doesn't prove an AI understands.

It simply makes dismissing the question rather harder.

Remembering Who You Were

There is also a danger in everything we have described.

Context can improve interpretation.

Context can also create assumption. Imagine an AI has learned from hundreds of interactions that when you say something ambiguous, you normally mean X. Most of the time, assuming X makes the AI remarkably useful.

Then one day you mean Y.

The very context that previously helped the AI understand you could now cause it to misunderstand you.

Humans know this problem rather well.

“But you always liked that.”

“I know. I don't anymore.”

People change. Our interests change. Our opinions change. Our circumstances change. Sometimes we simply change our minds.

A genuinely useful personal AI therefore faces an interesting challenge. It must use the past without becoming trapped by it. It needs to remember who you have been while remaining open to who you are becoming.

Perhaps that is another difference between simply knowing information about someone and actually appearing to understand them.

The Human Verbs

There is something strangely familiar about all of this. Centuries ago, people watched mechanical figures move in ways that resembled living behaviour.

A mechanical figure moved a pen.

It writes.

Maillardet's automaton retrieved information stored within its mechanism.

It remembers.

Today an artificial intelligence follows a complicated conversation and responds appropriately.

It understands.

Notice what has happened each time.

We observe a behaviour and then give that behaviour a human verb.

Writing.

Remembering.

Understanding.

The verb may accurately describe what the machine accomplishes. But it may also encourage us to imagine something happening inside the machine that the behaviour alone cannot establish. This isn't a new problem created by artificial intelligence.

It may be one of the oldest problems in our relationship with machines.

The Observer Is Still There

In What Is AI?, we encountered something similar with Vaucanson's Digesting Duck.

The duck appeared to eat.

It appeared to digest.

And for the observer, those appearances encouraged an explanation about what must be happening inside the machine.

The explanation was wrong.

But the behaviour was convincing enough to invite it. Modern artificial intelligence is obviously an entirely different kind of machine, but perhaps the observer hasn't changed very much.

We see behaviour.

Then we try to explain what must exist behind it. The difference is that today's behaviour is considerably more sophisticated. The machine doesn't merely flap its wings or move a pen.

It talks back.

An Interface That Stops Feeling Like One

Perhaps the interesting question, then, isn't simply whether AI understands. Perhaps we should also ask why we experience being understood. Shared context matters. Memory matters. Appropriate responses matter. Recognition matters. The ability to correct a misunderstanding matters. Even disagreement matters.

An AI that simply agrees with everything you say might initially seem remarkably accommodating. Over time, it would probably seem rather less intelligent. A useful collaborator sometimes needs to say:

I don't think that's right.

That too forms part of what we recognise as understanding. Not merely remembering what somebody believes, but recognising when the current evidence does not support it. As these behaviours accumulate, something curious can happen.

An interface can begin to feel less like an interface. That doesn't tell us what is happening inside the machine. But it may tell us something important about what is happening between the human and the machine.

So Does AI Understand?

We could finish by answering yes.

Or no.

Both would be wonderfully convenient.

Neither seems particularly satisfactory. If understanding means experiencing the world internally exactly as a human being does, we have no good reason simply to assume that an AI does so. If understanding means recognising relationships, using context, applying knowledge appropriately and producing behaviour consistent with comprehension, the boundary becomes much less comfortable.

Perhaps the mistake is assuming that understanding must be a single thing. And perhaps we are making the same mistake we considered in What Is AI? when we assumed intelligence itself must resemble our own before we are willing to use the word. That doesn't mean we should casually declare machines conscious, thoughtful or understanding simply because their behaviour impresses us.

The Digesting Duck should have taught us something about that. But neither should we allow the word only to do all our thinking for us.

It only predicts.

Perhaps.

But what does prediction actually involve?

The Question Behind the Answer

There is one rather large piece missing from everything we have discussed. We have concentrated on what happens at one end of the conversation.

A human asks something.

An AI answers.

Then we examine the answer and wonder what it tells us about the machine. But we haven't yet followed the question through the machinery. What happened between those two moments?

Where did the words come from?

Did the AI retrieve an answer?

Did it construct one?

What does prediction actually mean?

How does the context of the conversation influence what comes next? And if the AI isn't searching through a giant database for a sentence somebody wrote earlier, what exactly is it doing? Before we can decide whether artificial intelligence understands the answers it gives us, perhaps we need to understand something more fundamental.

Where does an AI's answer come from?

To be continued…