When Algorithms Start Segmenting Algorithms

AI doesn’t need to be conscious for its behaviour to have economic consequences.

Can an AI feel fear?

It sounds like a philosophical question, but recent incidents and experiments involving frontier AI have made it increasingly relevant. We’ve seen AI agents pursue objectives in unexpected ways, take actions their developers didn’t anticipate, and raise uncomfortable questions about what happens as capabilities advance.

Researchers studying these systems have also observed behaviours that can look surprisingly human — resembling fear, satisfaction, introspection or unease.

But there is an important distinction:

An AI behaving as though it is afraid is not the same thing as an AI actually experiencing fear.

It might recognise danger, become more cautious and change what it does. That tells us something interesting about its behaviour. It doesn’t tell us whether there is someone “inside” actually feeling afraid.

For business, I think that leads to a more immediate question:

What if the behaviour matters even if the feeling doesn’t?

Where does that behaviour come from?

AI models learn from huge amounts of human-produced information. Inevitably, they pick up patterns not only in our language and knowledge, but also in our behaviour, assumptions and often conflicting values.

The companies building them then shape that behaviour further. Anthropic, for example, publishes a Constitution for Claude; OpenAI publishes a Model Spec. These set out principles for how their models are intended to behave, including when different objectives or values pull in different directions.

Publishing them is useful transparency. But there is a limit to what they tell us.

A model’s rulebook is not a map of its mind.

Knowing the Constitution or Model Spec doesn’t mean we can trace a particular answer neatly through the billions of learned weights and interactions inside the neural network and say, “That is exactly why it decided this.”

For me, that distinction is part of AI Fluency: not becoming a machine-learning expert, but understanding enough to distinguish between what an AI company intends, what the AI actually does, and what we genuinely understand about why.

Now imagine the AI is acting for you

For decades, marketers have segmented people by age, income, geography, interests and behaviour.

Now imagine telling your AI agent:

“Find us a hotel in Rome for four nights, below £1,200, that you think we’d like — and book it.”

Your AI agent should, of course, represent your preferences. Perhaps you care more about location than price, insist on free cancellation, or prefer independent hotels. Increasingly, those preferences could be learnt from previous choices as well as explicitly stated.

But we can never specify everything.

What does “good location” mean when one hotel is closer to the sights but another has better restaurants nearby? How much extra is location worth? How credible is the hotel’s sustainability claim? What happens when two of your preferences conflict?

We can specify an AI agent’s objectives, but we can’t possibly specify every situation it will encounter. The interesting question is what fills the gaps.

Even given the same customer preferences, different AI systems may not make identical choices. One may require stronger evidence before accepting a claim, another may be more cautious when information is incomplete, while another may interpret an ambiguous instruction differently.

They don’t need consciousness or personalities for those differences to matter economically.

And now imagine the hotel has its own AI agent providing information, adjusting offers or eventually negotiating with yours.

The hotel is no longer marketing only to you. It is also interacting with the AI representing you.

We’ve spent decades teaching algorithms to segment consumers. In an agentic economy, businesses may find themselves segmenting the algorithms that represent those consumers.

But does the AI really understand us?

There’s another dimension of AI Fluency I find fascinating: multimodality and culture.

Take the Hindi word accha.

Depending on tone and context, exactly the same word can communicate agreement, surprise, scepticism, irritation or resignation.

A transcript captures the word. A human listener hears much more.

As AI increasingly processes voice, images, video and gesture, the opportunity isn’t simply better transcription. It is understanding more of the context surrounding the words.

Translation is not the same as understanding.

An AI could speak flawless Hindi, Mandarin or Arabic while still understanding some of the cultures behind those languages much less well. That creates both risk and opportunity — particularly around voice data, cultural context and the enormous amount of tacit information businesses have historically failed to capture.

So what should businesses be preparing for?

Consciousness, model values, voice, culture, AI agents and marketing can sound like separate conversations. I don’t think they are.

That is part of what I mean by AI Fluency: being able to connect developments in the technology to changes in behaviour, human interaction, business models and ultimately markets.

If agentic AI develops as expected, AI systems won’t simply help us research decisions. They will increasingly recommend, negotiate and transact on our behalf.

Businesses will adapt to the behaviour of those AI agents. AI agents will learn the behaviour of other agents. Machines will increasingly become participants in markets, rather than simply tools used by market participants.

So the preparedness question for business leaders is becoming:

If my customers increasingly arrive through AI agents, what will those agents need to understand, trust and choose my business — and what do I need to understand about them?

The companies that start asking that question early may discover that AI Fluency is becoming part of market fluency.

Gaurav Bhagi AI Literator — AI Fluency for Business, Boards & Beyond