There Is No Big Red Button: The Real Problem with AI Control

Why human oversight, accountability, and the power to challenge AI decisions matter more than a simple shutdown switch

Author: Lodrick Wabwire Odo | AI, Data & Digital Governance

Shaping Africa’s Future with Evidence, Equity, and Innovation for Impact

Affiliation: African Centre for Social and Population Research
Publication type: ACSPR Research Commentary
Publisher: African Centre for Social and Population Research
Publication date: October 2026
DOI: https://doi.org/10.67810/acspr-rc-0001

When people worry that artificial intelligence might one day take control, the reassuring answer is often simple:

“Humans built it. If it becomes dangerous, we’ll turn it off.”

That answer works when we imagine one machine with one power button. Real life is more complicated. AI can be connected to the services we use, the information we see, the decisions that affect us, and, increasingly, devices that interact with our bodies. Turning off one system may be possible. Understanding its influence and deciding who has the authority to stop it - requires much more thought.

There is no single big red button for every AI system shaping modern life. And even where a shutdown option exists, it is not enough on its own.

Recent scholarship on AI oversight makes a similar point: modern AI risks often emerge through connected workflows, cumulative decisions, institutional dependency, and unclear lines of responsibility rather than through one dramatic moment that can be solved by an emergency switch alone. Effective control therefore requires continuous oversight, contestability, and clear accountability across the AI lifecycle (Bahidika, 2026; Manheim & Homewood, 2025; Van De Sande et al., 2026; Zhu et al., 2026).

AI control does not always fail through a visible takeover. Sometimes control weakens gradually.

Imagine a hospital using AI to identify patients who need urgent follow-up. At first, clinicians carefully review every recommendation. As workloads increase, they come to rely on the system and check fewer cases. Eventually, its recommendations largely determine who is contacted quickly and who must wait.

The AI has not declared itself in charge. Yet if staff cannot understand an error, override a recommendation, or reach patients the system missed, human control has become weaker.

Now imagine a government using AI to screen applications for a public benefit. It may process cases faster, but what happens when an eligible person is rejected because a record is incomplete? That person needs an explanation and a workable way to appeal. Having a human sign off on thousands of recommendations offers little protection if that person cannot genuinely review them.

These examples are hypothetical, but the question is practical:

This is where the idea of contestability becomes important. Contestability means that affected people should be able to question, challenge, appeal, and seek correction when an AI-influenced decision affects them. Without contestability, human oversight can become symbolic rather than meaningful (Alfrink et al., 2022; Koulu, 2020; Laux, 2023).

In 2012, a software problem at the trading firm Knight Capital sent erroneous orders into financial markets. The company initially reported a realized pre-tax loss of approximately US$440 million, and later annual reporting described trading losses of US$457.6 million linked to the August 1, 2012 incident. This was automated trading software, not an AI system attempting to seize control. But it illustrates how quickly a connected system can cause harm before people contain an error.

As AI systems become able to take sequences of actions, using software, sending messages, operating connected tools, or supporting decisions across multiple platforms, the time available for meaningful intervention becomes even more important. A person cannot effectively supervise actions they cannot see until after the consequences have spread.

That is why “a human is in the loop” should be more than a comforting phrase. The human needs enough information, authority, and time to intervene.

This matters because human operators can experience automation bias, reduced vigilance, and cognitive overload when they are expected to monitor complex systems passively and intervene only when something goes wrong (Gaube et al., 2026; Sterz et al., 2024). When oversight depends on a person noticing a problem at the last moment, the system may already have failed by design.

In other words, the real issue is not simply whether a human is present. The issue is whether the human has:

Without these conditions, a human reviewer can become what some scholars describe as a liability buffer: a person placed near the system to absorb responsibility without having real power to prevent harm (Gaube et al., 2026; Van De Sande et al., 2026).

AI does not gain control merely by becoming clever.

A company might let an AI assistant answer customer questions. Later, it might let the same system change accounts, issue refunds, recommend loans, screen applicants, send messages, or make purchasing decisions. Each added permission increases its reach.

If an error occurs, the important questions are concrete:

The same questions become more serious when AI is used in public services. People may have little choice about whether to interact with a government system. They should still be able to find out when AI has influenced a consequential decision, challenge mistakes, and reach a person with the power to correct them.

Control also concerns who owns and operates the systems. If a small number of organisations provide AI tools on which schools, hospitals, businesses, and governments depend, their choices can affect many people. Public oversight and accountability cannot end at the door of the technology provider.

This is why AI governance must align authority, responsibility, and rights. If an institution gives an AI system operational power, it must also define who remains accountable, who has the authority to intervene, and what rights affected people have when decisions cause harm (Novelli et al., 2023; Sun, 2026).

For African countries, this debate is urgent. AI is increasingly being discussed in health, education, finance, agriculture, public administration, digital identity systems, research, and service delivery. These tools may help governments and institutions work faster, reach underserved populations, and improve planning. But they can also create new risks if they are deployed without adequate oversight, data protection, accountability, and appeal mechanisms.

The question for Africa is not simply whether AI can improve services. It is whether institutions have the capacity to monitor AI systems, explain their decisions, protect citizens’ data, correct harm, and ensure that technology does not deepen existing inequalities.

In countries where public services are already under pressure, weak oversight can make automated errors more difficult to detect and correct. If an AI system affects access to healthcare, credit, social protection, education, employment, or public records, people must have a clear way to understand and challenge the outcome.

This is especially important because many African societies already face challenges related to administrative capacity, digital exclusion, weak data systems, limited access to redress, and unequal service delivery. If AI systems are introduced into these environments without strong safeguards, they may not only reproduce existing inequalities but also make them harder to contest.

AI governance is therefore not only a technical issue. It is a public accountability issue.

The good news is that practical AI oversight is not imaginary. Frameworks already exist.

The US National Institute of Standards and Technology’s AI Risk Management Framework is intended to help organisations incorporate trustworthiness into the design, development, use, and evaluation of AI systems. Its AI RMF Core notes that documentation can improve human review and accountability, and that processes for human oversight should be defined, assessed, and documented.

In other words, serious AI governance is not about trusting a switch. It is about building systems where oversight is planned, recorded, tested, and accountable.

The European Union’s AI Act provides another example. Article 14 requires high-risk AI systems to be designed to allow human oversight during operation, with the aim of minimizing risks to health, safety, and fundamental rights.

That matters because it shows that human oversight can be designed into systems before harm occurs. The lesson is clear: the best safeguard is not panic after failure, but governance before deployment.

Beyond formal regulation, researchers also emphasize contestability by design: AI systems should be built so that affected people can question, challenge, and seek review of decisions. Oversight should therefore include not only internal monitoring, but also meaningful pathways for appeal, explanation, and correction (Alfrink et al., 2022; Koulu, 2020; Laux, 2023).

This is important because oversight should not be reduced to a person sitting at the end of a decision chain approving what the system has already decided. Meaningful oversight requires the power to pause, question, reverse, escalate, and correct.

A shutdown button can be useful, but it answers only one question:

Modern AI governance requires a broader set of controls. The real challenge is not only stopping a system after harm begins,  but designing systems so that harm is less likely to occur, easier to detect, and easier to correct.

 1.     Emergency shutdown is not enough; A weak control model relies on emergency shutdown. A stronger control model builds continuous monitoring and staged intervention.

 2.     Human review must be meaningful; A weak control model assumes that having a human reviewer is enough. A stronger control model gives human reviewers information, authority, time, and institutional support.

 3.     Accountability must be institutional; A weak control model treats accountability as an individual burden. A stronger control model defines institutional responsibility across the AI lifecycle.

4. Decisions must be contestable; A weak control model allows opaque automated decisions. A stronger control model requires explanation, records, appeal, and correction.

 

5. Permissions must be limited; A weak control model gives AI broad permissions. A stronger control model limits permissions according to risk and reversibility.

 

6. Governance must come before harm; A weak control model responds after harm occurs. A stronger control model tests, documents, and governs before deployment.

This shift matters because AI systems may operate across complex workflows. In safety-critical domains, one action may seem low-risk on its own, but a sequence of actions can produce serious consequences. Effective control therefore requires intermediate options between letting the system run and shutting it down completely.

In technical governance terms, this points to the need for runtime intervenability: systems should allow proportionate human intervention before harm escalates, rather than offering only two extreme choices, let the system continue or shut everything down completely (Aarab, 2026; Herrmann, 2026; Tao, 2025).

Brain–computer interfaces bring the question of control even closer to home.

Researchers are studying implants that translate certain brain signals into commands, allowing people with paralysis to control devices. In one study, three participants with tetraplegia used an intracortical brain–computer interface to control an unmodified commercial tablet computer (Hochberg et al., 2018). Other research has demonstrated speech neuroprostheses that decode neural activity associated with attempted speech into text or sound, offering possible communication pathways for people with severe paralysis or speech impairment (Willett et al., 2023).

These are promising medical advances. They are not evidence that today’s brain chips can read every private thought or allow AI to take over a person’s mind.

Still, these technologies raise questions we should address early:

The point is not to frighten people away from a technology that may restore independence. It is to ensure that the people using it retain meaningful choice and that safeguards grow alongside its capabilities.

Brain–computer interfaces make the wider AI governance question more personal: when technology becomes closer to the body, the right to consent, disconnect, understand, and control becomes even more important.

Keeping humans in control requires more than one emergency option. It requires a plan.

A serious AI control plan should answer at least seven questions:

These are the questions that turn the idea of oversight into a practical duty.

They also help avoid a common problem in AI governance: placing responsibility on frontline staff or ordinary users while the real design and deployment power sits elsewhere. Effective governance must therefore distribute accountability across the system from developers and vendors to institutions, regulators, managers, and public authorities (Abulibdeh et al., 2026; Bogiatzis-Gibbons, 2024; Novelli et al., 2023).

For governments: AI systems used in public services should have clear rules on transparency, appeal, human review, data protection, and accountability. People should not be locked out of essential services by systems they cannot understand or challenge. This is especially important in high-stakes sectors such as healthcare, social protection, finance, education, and public administration, where automated decisions can affect rights, access, and welfare (Abulibdeh et al., 2026; Bogiatzis-Gibbons, 2024).

For companies: AI should not be given wider permissions without testing, monitoring, documentation, and emergency controls. Expanding what an AI system can do should come with stronger responsibility, not weaker oversight. Companies should not treat AI failures as isolated technical accidents when those failures result from organisational choices about permissions, monitoring, incentives, and deployment speed.

For health and research institutions: AI and brain–computer interface technologies should protect consent, privacy, safety, and the user’s right to limit or disconnect systems. Medical innovation should strengthen human dignity, not weaken autonomy. In health settings, “human in the loop” should mean more than a clinician being present; it should mean meaningful authority, explainability, escalation pathways, and institutional responsibility (Van De Sande et al., 2026).

For citizens: People should ask when AI is being used, how decisions are made, what data is collected, and how mistakes can be corrected. Public trust depends on the ability to question and challenge consequential decisions.

For researchers and civil society: More attention is needed on AI oversight, digital rights, data protection, neurotechnology ethics, institutional accountability, and the social impacts of automated decision-making, especially in contexts where institutions may lack strong accountability systems. Researchers should also examine whether oversight mechanisms actually work in practice, not only whether they appear in policies.

For African institutions: AI adoption should be matched with governance capacity. Before deploying AI in high-impact areas, institutions should ask whether they have the data systems, legal safeguards, human expertise, appeal pathways, and operational capacity needed to protect the public.

When a service you use adopts AI, you can ask:

If the answer to these questions is unclear, that is a red flag.

These questions matter because AI governance should not be reserved for engineers, lawyers, or policymakers alone. Ordinary people should have the right to understand when automated systems affect their lives.

Could a much more capable AI someday become difficult for humanity to control? Possibly. We cannot confidently predict whether or when that will happen.

We can, however, recognise a more immediate path by which control weakens: give systems greater authority, depend on them heavily, make their decisions hard to question, and wait until something goes wrong to decide who was responsible.

We can choose another path.

We can limit AI’s permissions, test it before expanding its role, monitor its actions, protect sensitive data, and preserve people’s power to intervene. We can require institutions to explain consequential decisions and correct mistakes. We can design systems with contestability, auditability, and proportional intervention from the start.

So, should we panic about AI taking control?

No.

But should we accept “we can always switch it off” as a complete answer?

Absolutely not.

The closer AI gets to our decisions, our services, and even our bodies, the more important it becomes to preserve our ability to understand, challenge, and stop what it does.

How to cite this article:

Odo, L. W. (2026). There Is No Big Red Button: The Real Problem With AI Control. African Centre for Social and Population Research. https://doi.org/10.67810/acspr-rc-0001

Aarab, H. (2026). The legitimacy layer: A runtime governance architecture for human oversight in AI-assisted critical decision systems. 2026 1st International Conference on Human Centric Artificial Intelligence (ICHCAI), 1–10. https://doi.org/10.1109/ICHCAI70183.2026.11607581

Abulibdeh, R., Agyemang, G. O., Celi, L., Gorijavolu, R., Kalema, N., Kleinlein, R., Madapati, K., Salarikia, S. R., & Youssef, A. (2026). Who’s really in the loop? Rethinking oversight in AI-assisted health care. The Lancet, 407(10545), 2340–2344. https://doi.org/10.1016/S0140-6736(26)00204-7

Alfrink, K., Keller, I., Kortuem, G., & Doorn, N. (2022). Contestable AI by design: Towards a framework. Minds and Machines, 33, 613–639. https://doi.org/10.1007/s11023-022-09611-z

Bahidika, O. (2026). Agentic accountability: “The buck stops where?” Ethical frameworks for human oversight of autonomous AI systems. International Journal of Advanced Computer Science and Applications. https://doi.org/10.14569/IJACSA.2026.0170502

Bogiatzis-Gibbons, D. J. (2024). Beyond individual accountability: (Re-)asserting democratic control of AI. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3630106.3658541

European Union. (2024). Artificial Intelligence Act, Article 14: Human oversight.

Gaube, S., Langer, M., Miller, T., Baum, K., Dachselt, R., Feit, A., Gadiraju, U., Kaur, H., Keane, M. T., Landers, R., Laux, J., Liao, Q., Lim, B., Onnasch, L., Schrills, T., Sonenberg, L., Tan, C., Tintarev, N., Xiao, Z., & Zhang, H.-W. (2026). Keeping an eye on AI: A framework for effective human oversight of AI systems. arXiv. https://doi.org/10.48550/arXiv.2605.16278

Herrmann, T. (2026). Intervenability as a design requirement for autonomy and oversight within human-centered AI. arXiv. https://doi.org/10.1007/978-3-031-83512-4_9

Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., Haddadin, S., Liu, J., Cash, S. S., van der Smagt, P., & Donoghue, J. P. (2018). Cortical control of a tablet computer by people with paralysis. PLOS ONE, 13(11), e0204566. https://doi.org/10.1371/journal.pone.0204566

Knight Capital Group. (2012). Knight Capital Group provides update regarding August 1, 2012 disruption to routing in NYSE-listed securities. U.S. Securities and Exchange Commission filing.

Koulu, R. (2020). Proceduralizing control and discretion: Human oversight in artificial intelligence policy. Maastricht Journal of European and Comparative Law, 27, 720–735. https://doi.org/10.1177/1023263X20978649

Laux, J. (2023). Institutionalised distrust and human oversight of artificial intelligence: Towards a democratic design of AI governance under the European Union AI Act. AI & Society, 39, 2853–2866. https://doi.org/10.1007/s00146-023-01777-z

Manheim, D., & Homewood, A. (2025). Limits of safe AI deployment: Differentiating oversight and control. arXiv. https://doi.org/10.48550/arXiv.2507.03525

National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework 1.0. https://doi.org/10.6028/NIST.AI.100-1

National Institute of Standards and Technology. (2023). AI Risk Management Framework Core.

Novelli, C., Taddeo, M., & Floridi, L. (2023). Accountability in artificial intelligence: What it is and how it works. AI & Society. https://doi.org/10.1007/s00146-023-01635-y

Sterz, S., Baum, K., Biewer, S., Hermanns, H., Lauber-Rönsberg, A., Meinel, P., & Langer, M. (2024). On the quest for effectiveness in human oversight: Interdisciplinary perspectives. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3630106.3659051

Sun, F. (2026). Authority and responsibility allocation in human–AI collaborative decision-making: Governance mechanisms for enterprises. ICCK Transactions on Systems Safety and Reliability. https://doi.org/10.62762/TSSR.2026.930520

Tao, Y. (2025). The decision path to control AI risks completely: Fundamental control mechanisms for AI governance. arXiv. https://doi.org/10.48550/arXiv.2512.04489

Van De Sande, D., Economou-Zavlanos, N. J., & Van Genderen, M. V. (2026). Meaningful oversight of medical AI beyond human in the loop. NPJ Digital Medicine, 9. https://doi.org/10.1038/s41746-026-02971-1

Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E. Y., Kamdar, F., Glasser, M. F., Hochberg, L. R., Druckmann, S., Shenoy, K. V., & Henderson, J. M. (2023). A high-performance speech neuroprosthesis. Nature, 620, 1031–1036. https://doi.org/10.1038/s41586-023-06377-x

Zhu, L., Lu, Q., Ding, M.-M., Lee, S. U., & Wang, C. (2026). Designing meaningful human oversight in AI. AI and Ethics, 6. https://doi.org/10.1007/s43681-026-01147-7