Who Gets to Say “Stop” in the AI Race?

Government needs technical competence

As a software engineer, I would be uncomfortable with people regulating a system they cannot meaningfully question.

An oversight team needs to understand what an evaluation measures, what it misses, and how deployment changes the risk. Otherwise, a polished presentation can pass for evidence.

But “government does not understand technology” is too broad. Government includes researchers and engineers, as well as elected officials.

The UK’s AI Security Institute publishes technical evaluations of frontier systems, including autonomous software work and cybersecurity capabilities. It also explicitly acknowledges that its evaluations do not capture every factor affecting real-world outcomes. That combination of technical work and clear limits is useful.

AISI’s Frontier AI Trends Report

Understanding a risk and enforcing a response

In the United States, NIST’s Center for AI Standards and Innovation describes a remit that includes technical evaluations and voluntary agreements with developers. Those activities demonstrate technical capacity; they do not, by themselves, establish a comprehensive enforcement system.

The distinction matters. Understanding a risk and having the authority to require a response are separate capabilities. Effective oversight needs both.

NIST’s Center for AI Standards and Innovation

Engineers need a voice and independence

Engineers should be deeply involved. We need people who can inspect the system, reproduce failures, challenge assumptions, and explain why a reassuring benchmark might miss an important problem.

But technical expertise does not remove a conflict of interest.

An engineer employed by a frontier lab works inside an organisation with launch schedules, commercial commitments, and management authority. Being qualified to identify a problem does not necessarily mean being empowered to stop a release.

Nor does knowing how to build a model give someone the sole right to decide how much risk other people should accept.

The people affected include workers, customers, smaller businesses, and communities that never agreed to participate in the experiment. Their interests deserve representation too.

Some AI employees have raised this issue themselves. The June 2024 Right to Warn letter argued that financial incentives and confidentiality restrictions could obstruct effective oversight. Its signatories called for protected reporting channels and safeguards against retaliation. These are the signatories’ claims and proposals, rather than a finding that every AI company suppresses concerns.

My takeaway is practical: if we want engineers to be part of the accountability system, their ability to speak cannot depend entirely on their employer’s comfort with the message.

A Right to Warn about Advanced Artificial Intelligence

Follow the money carefully

The financial scale deserves scrutiny. It also deserves accurate language.

A company valuation, an infrastructure commitment, and money already spent are different things. Adding them together produces a dramatic number with very little explanatory value.

For example, OpenAI’s January 2025 Stargate announcement described an intention to invest $500 billion over four years in US AI infrastructure. That was a multiyear investment plan, not a statement that $500 billion had already changed hands.

OpenAI’s original Stargate announcement

The relationships behind the numbers

The relationships behind the numbers are more revealing.

The FTC’s January 2025 staff report examined partnerships between major cloud providers and AI developers. It identified arrangements requiring developers to spend substantial portions of their partners’ investments on those partners’ cloud services, alongside equity, revenue-sharing, and certain control rights. It also described potential competition concerns, including higher switching costs.

The report reflects information available to staff through September 2024, supplemented by public information through January 2025. These findings describe arrangements documented at the time of the report; individual partnerships may have changed since then.

That report does not establish that companies compromised safety. It shows how closely financing, infrastructure, and commercial interests can be connected.

My concern is what happens when a serious technical finding collides with those commitments.

Would a release still be delayed? Who makes that decision? Would the evidence reach anyone outside the organisation?

Investors are entitled to pursue returns. The public is entitled to an oversight process that can reach an inconvenient conclusion.

We do not need to prove that every stakeholder cares only about money. We need arrangements that remain credible when money and safety point in different directions.

FTC AI partnerships and investments study

Independence has to mean something operational

Calling an evaluator “independent” tells me very little until I know the terms of the work.

Can they select their own tests? Do they have enough time? Can they inspect relevant records? Can they report a serious finding without the company rewriting the conclusion?

A research preprint posted to arXiv on January 17, 2026, on external frontier-model evaluations identifies limited access, information, and evaluation time as obstacles to rigorous testing. It also recognises that broader access brings security risks of its own. Access needs to be sufficient and controlled.

This is familiar engineering territory. A review is only as useful as the evidence available to the reviewer.

Research preprint: Expanding External Access to Frontier AI Models for Dangerous Capability Evaluations

What I would want an oversight system to do

My proposed oversight model would have five parts.

• Publicly accountable authorities set the boundaries. Define which risks trigger scrutiny and what evidence is required. Give affected people a voice in those decisions.
• Independent technical teams investigate. Provide secure access, adequate time, and funding that does not depend on pleasing the company under review.
• Engineers have protected escalation channels. Serious concerns should be able to reach qualified investigators outside the management chain.
• Enforcement has consequences. For defined, serious risks, authorities should be able to require mitigation, restrict deployment, or order a temporary pause, with written reasons and a route to challenge the decision.
• The overseers are accountable too. Publish methods and decision rationales where security and privacy permit. Disclose conflicts and review whether the rules actually work.

These are recommendations, not a description of a complete system already in place.

Government can have conflicts too

Moving authority outside a company does not make incentives disappear.

A government may also want investment, jobs, or a national advantage. I would design oversight on the assumption that pressure to accelerate can come from public officials as well as executives.

Technical findings therefore need protection from political interference. Decisions need scrutiny. Restrictions need clear reasons and review dates.

The rules should also be proportionate. A small team building a narrow business tool should not automatically face the same requirements as a lab developing a highly capable model with broad deployment.

Otherwise, oversight could become an expensive barrier that only the largest companies can afford. A system intended to constrain concentrated power should be evaluated for whether it strengthens that concentration.

International coordination matters as well. Shared testing practices and incident reporting could reduce gaps between jurisdictions. A single government cannot settle a global competition on its own, but imperfect coordination is still worth pursuing.

The real test comes when the answer is inconvenient

I want engineers involved because technical competence matters. I want independent scrutiny because competence alone does not resolve conflicting incentives. And I want public accountability because these decisions affect people beyond the companies making them.

For engineers and businesses deciding which AI providers to trust, I would look past the safety statement and ask about the process.

Who saw the evidence? What could they inspect? What happens if they disagree? Who has the authority to require a change?

An oversight system earns credibility when a well-supported finding can change a release decision, even when that decision is expensive.

That is the standard I think this industry should be working toward.