FOUNDATION INVESTIGATION | PEOPLE IN CONTROL
AI is becoming better at anticipating what people want and guiding what they do next. The challenge is keeping the convenience while preserving judgement, agency and the ability to say no.
By Andrew McDonald · Immortal AI
Most people will not hand control of their lives to an AI in one dramatic decision.
It will happen in smaller ways.
The route it recommends.
The email it writes.
The candidate it ranks first.
The news it puts in front of you.
The answer it gives before you have worked through the question yourself.
Each decision may be sensible. Each saves time.
Together they raise a larger question:
Can AI help us without gradually teaching us to stop deciding for ourselves?
Convenience is how control changes hands
People usually give technology authority because it works.
Navigation apps replaced paper maps because they are faster. Recommendation systems save us from searching through thousands of choices. Generative AI can turn an hour of drafting into minutes.
There is nothing inherently wrong with that.
The problem begins when assistance quietly becomes default judgement.
A recommendation is easy to accept because rejecting it requires effort. The more often the system is right, the less reason there appears to be to inspect the next answer closely.
That is how a useful tool can begin to shape behaviour without anyone consciously deciding to surrender control.
People in the loop are not enough
“A person remains in control” has become one of the standard reassurances around AI.
But being present is not the same as being capable of intervening.
Effective oversight requires knowledge, time, authority and an interface that makes intervention realistic.
The European Union’s AI Act recognises this directly. For high-risk systems, Article 14 requires effective oversight by natural persons and says those people should understand the system’s capabilities and limits, monitor for unexpected performance, remain alert to automation bias, interpret outputs and be able to intervene or stop the system where appropriate. EU AI Act, Article 14.
That is much stronger than putting a person at the end of a workflow and calling the process supervised.
We explored the same distinction in AI Rejected You. Who Is to Blame?: meaningful oversight exists only when somebody understands enough and has enough authority to change the outcome.
Automation bias is a human weakness, not a machine feature
AI does not need to force anybody to follow its recommendation.
People can over-trust it themselves.
This tendency is often called automation bias: relying too heavily on an automated output simply because it came from a system assumed to be competent.
The EU AI Act specifically warns overseers of high-risk systems about this risk. NIST’s AI Risk Management Framework likewise says organisations need clearly defined roles for people and AI, documented oversight processes and people who understand system performance and trustworthiness. NIST AI RMF Core.
The danger grows as systems become more capable.
A poor tool invites scepticism.
A very good tool can make scepticism feel inefficient.
Control also depends on what the system knows about you
An assistant that knows nothing about you has limited power to personalise its advice.
An assistant that remembers your history, preferences, fears, relationships and habits can become much more useful.
It can also become more persuasive.
That is why control cannot be separated from privacy.
In AI Knows What You Fear, Want and Regret, we examined how intimate information can improve the system’s ability to anticipate what a person will respond to. The same personalisation that makes advice feel relevant can make influence harder to recognise.
A recommendation that appears to understand you can carry more weight than a generic one.
That does not make personalisation wrong.
It makes transparency about why the recommendation exists more important.
A system should help people think, not replace the need to think
There is a difference between reducing unnecessary effort and removing judgement.
AI can summarise a long report so a person can focus on the important parts.
It can also summarise it so aggressively that the person never sees the evidence that would have changed their mind.
It can suggest a medical question to ask a doctor.
It can also provide such a confident answer that the person decides the doctor is unnecessary.
It can help a manager compare candidates.
It can also become the ranking the manager stops questioning.
This is one reason confident error matters. The AI Answer Was Completely Wrong. Why Did It Sound So Convincing? examines what happens when fluency itself begins to act as a credibility signal.
Control requires keeping enough friction in consequential decisions for people to notice when they should stop and think.
The strongest governance frameworks put responsibility back on organisations
Good AI governance does not tell users simply to be more careful.
It asks the organisations building and deploying systems to create conditions in which people can remain meaningfully in control.
The OECD’s AI Principles call for safeguards that preserve human agency and oversight and say organisations and individuals developing, deploying or operating AI should be accountable for proper functioning according to their roles. OECD: Human-centred values and fairness. OECD: Accountability.
NIST similarly places responsibility on organisations to define oversight roles and has executive leadership take responsibility for AI risk decisions. NIST AI RMF Core.
Australia’s responsible AI implementation guidance also emphasises accountability, impact assessment, transparency, testing and maintaining oversight as practical organisational responsibilities. Australian Guidance for AI Adoption summary.
This is the right direction.
The burden should not fall on the least informed person in the system.
Control must include the ability to refuse
A person is not meaningfully in control when declining AI creates an unreasonable penalty.
A worker who can technically ignore an automated recommendation but knows their performance will be questioned if they do is not exercising free judgement.
A customer who can opt out only by abandoning an essential service has little practical choice.
A child interacting with a highly relational chatbot may not understand the commercial incentives behind the interaction at all.
As we found in When AI Becomes the Only One Who Listens, vulnerability makes formal choice an especially weak safeguard.
Agency needs alternatives.
People should be able to understand when AI is materially shaping an outcome, challenge it when the stakes are high and access a person with the authority to make a different decision.
We also need to preserve capability
There is another form of control that is easier to overlook.
What happens when people lose the skills needed to take over?
A pilot cannot provide meaningful backup if automation has allowed critical flying skills to decay. The same principle can apply to analysts, managers, students, clinicians, writers and ordinary users making everyday decisions.
Using AI does not automatically make people less capable.
Used well, it can expose people to ideas, challenge assumptions and accelerate learning.
But systems designed primarily to remove effort can also remove practice.
That means good implementation should ask not only whether AI makes a task faster, but what skill the person still needs when the system is unavailable, wrong or working outside its intended conditions.
People in control is a design choice
We should not romanticise doing everything manually.
AI can remove drudgery, make expertise more accessible and help people make decisions with information they could never process alone.
The goal is not to keep people busy proving they can outperform a machine.
It is to keep responsibility, judgement and the ability to intervene where they matter.
Our Immortal AI Foundations ask who has the power, who bears the risk and who is responsible for the outcome. People in control is where those questions converge.
AI does not need to take control from us.
We can hand it over ourselves, one convenient decision at a time.
The question is not whether AI will become capable of thinking for us.
It is whether, when it can, we will still consider thinking for ourselves worth the effort.
Principal sources
- OECD AI Principle: Human-centred values and fairness
- OECD AI Principle: Accountability
- NIST AI Risk Management Framework Core
- NIST AI RMF: Human-AI Interaction
- EU Artificial Intelligence Act, Article 14
Editorial note: This article examines agency and oversight across different AI uses. The appropriate level of oversight depends on the stakes, system and context.
AI disclosure: Immortal AI uses AI-assisted research and drafting. Sources, claims, framing and final editorial decisions remain the responsibility of Immortal AI.
Leave a Reply