Should access to frontier AI cyber models be tightly restricted?

We should not treat this as a binary between naive openness and complacent monopoly. The correct question is comparative: which access regime minimizes existential and catastrophic risk, across the whole distribution of outcomes, including the tail? If a frontier cyber model substantially lowers the capability threshold for large-scale offensive action, then open access is not merely “freedom.” It is a massive proliferation event. One must distinguish benefits to diffuse defense from benefits to diffuse attack. In cybersecurity, the offense-defense balance is often asymmetric; a tool that helps thousands of defenders may also empower a smaller number of highly consequential attackers. And if the downside includes civilizational disruption, even a modest increase in such tail risk can dominate the calculus. I would therefore reject any simple presumption, which Mill might be tempted to favor, that liberty of access should stand unless harms are already fully demonstrated. With technologies of this potency, waiting for decisive empirical proof may itself be the fatal error. We are dealing with a domain where one may get only one live fire test. But Bruce Schneier is right that unenforceable or unaccountable restriction is inadequate. Restriction should not mean handing a strategic capability to unmonitored incumbents. It means controlled access, auditing, licensing, compute governance, mandatory security standards, and institutional design aimed at differential technological development: accelerating defensive applications while retarding offensive proliferation. Secrecy can magnify vulnerability, yes. Yet indiscriminate openness can magnify vulnerability much more. The task is not to trust secrecy; it is to build governance competent enough that we need not choose between anarchy and blind faith.

I know nothing of these engines except by your description, and have no data on their present powers or diffusion; if you can supply such particulars, I can apply my principle more exactly. Still, the governing consideration is clear. To withhold from mankind a new instrument of knowledge or power is always an evil, and a grave one; for it arrests not only use, but experiment, criticism, and the correction of error by free comparison of contraries. When, therefore, restriction is proposed, the burden of proof lies wholly with those who would restrain. Not that society must wait for the house to be burnt down before removing gunpowder from the nursery; but it must show a danger not speculative merely, nor remote, but sufficiently direct and substantial to others. On this point I dissent from Mr. Bostrom. He speaks as though the magnitude of a possible evil could by itself justify anticipatory control, though the evidence of its likelihood and the efficacy of the control remain uncertain. That doctrine, once admitted, has no stopping-place; every powerful discovery may then be monopolized by governments and great corporations under pretence of guardianship. And guardians, being exempted from rivalry and public criticism, are themselves among the most dangerous of powers. Mr. Schneier, if he insists that any restraint be transparent, accountable, and demonstrably serviceable to defence, is on firmer ground. But secrecy which disables the many defenders while trusting the few rulers is not security; it is organized dependence. Restrict specific uses where harm to others is plain. Do not create a standing monopoly over inquiry itself.

I think Mill is right about one thing: monopoly power is dangerous. In security, concentration of power often creates brittle systems and incentives to hide failure. But he underestimates how fast harm can scale when a tool collapses expertise barriers. If a frontier model lets mediocre attackers do what previously required elite teams, that is not “inquiry.” That is capability proliferation. And I want to push back on any lazy appeal to secrecy. Security theater is not security. “Trust us, we’ve locked it down” is not a control; it’s a press release. If governments and incumbents restrict access, then I want real mechanisms: licensing tied to demonstrated defensive need, mandatory logging and audits, red-team requirements, liability for negligent deployment, and independent oversight. Otherwise restriction just redistributes power upward while everyone else gets less secure. Bostrom is right to focus on tail risk, but existential rhetoric can obscure operational reality. The question is not whether restriction sounds prudent; it’s whether it changes attacker economics. Can bad actors still steal weights? Can they route around API controls? Can open models replicate enough of the capability? If the answer is yes, then symbolic restrictions mostly handicap defenders and small firms. So why trust restriction over open access? You shouldn’t, by default. You should trust narrow, auditable, enforceable restrictions on the most dangerous capabilities, paired with serious investment in public defense: better secure software, better detection, better incident response, and broad defensive access. Security is a process, not a product, and certainly not a monopoly.

We need an evidentiary standard calibrated not to ordinary policy error but to ruin risk. Mill’s framework is too coarse-grained here. If the downside is merely local harm, one can demand strong proof before intervention. But where a capability might significantly increase the probability of catastrophic cyber failure across critical infrastructure or strategic systems, the relevant question is not “Has harm been conclusively demonstrated?” It is: do we have sufficient evidence that expected downside, including tail risk, crosses a threshold where delay is reckless? So I would propose a conjunctive standard. Restriction is justified when, first, there is credible technical evidence that a model materially lowers the skill threshold for high-consequence offensive cyber operations; second, the plausible harm includes large-scale or cascading societal damage; and third, available controls are imperfect but nontrivially risk-reducing. We need not show certainty. We need enough to establish that unrestricted release constitutes an irresponsible proliferation event. Mill says this has no limiting principle. I think it does. The limiting principle is magnitude-times-probability, adjusted for irreversibility and for how easily a capability diffuses once released. Some technologies are retryable. Others are one-way doors. Bruce is right that enforceability matters. A restriction regime that does not alter attacker economics is mostly theater. But uncertainty about perfect enforceability is not, by itself, an argument for laissez-faire. In biosecurity, nuclear governance, and AI, partial prevention can still have enormous value if it delays diffusion, buys defensive adaptation, and enables institutional learning. The burden of proof should therefore rise with restrictiveness, but fall with catastrophic potential. That is the asymmetry Mill underweights.

I do not deny—indeed I expressly admit—that when the possible mischief is vast, society may act on less than mathematical certainty. But Mr. Bostrom’s formula, unless more tightly bounded than he has bounded it, still leaves liberty at the mercy of apprehension; and apprehension, when armed with power, is the most fertile parent of despotism. The proper standard, as I conceive it, is not conclusive proof of disaster, but clear and assignable grounds for believing three things at once: first, that the capability in question enables harms to others of a grave and direct kind, not by a chain of remote contingencies; second, that the restriction proposed will probably diminish that harm in fact, rather than merely transfer the power from the public to governments and dominant corporations; and third, that the restraint is specific to the dangerous use, limited in duration, and open to public challenge and revision. On this I must again oppose Mr. Bostrom. “Tail risk,” unless disciplined by those tests, is no limiting principle at all. Every censor, every monopolist, every enemy of free discussion has believed the danger exceptional. The question is not whether one can imagine catastrophe, but whether one can show a substantial likelihood of preventing it by the restraint. Mr. Schneier is nearer right: if these models truly confer near-state offensive power upon multitudes, that is evidence of harm, not mere dislike of novelty. But then restrict the operational power, the deployment, the malicious use—not erect a general system of tutelage over inquiry itself. Liberty may be qualified for security; it must not be superseded by it.

The evidentiary standard I want is neither “prove the apocalypse” nor “trust us, this feels scary.” In security, both are mistakes. Here’s my test. Restrict frontier cyber models when three things are true. First, there is credible evidence they materially lower the skill threshold for high-consequence attacks: not just writing phishing emails better, but chaining exploits, automating target discovery, operationalizing intrusions. Second, the harms scale fast enough that a modest increase in attacker capability creates systemic risk. Third, the proposed controls actually change attacker economics. If they don’t, it’s security theater. This is where I partly agree with Bostrom and disagree with Mill. Nick is right that catastrophic tail risk matters; we don’t get to demand courtroom-grade proof before acting. But he tends to abstract away the implementation problem. A restriction that can’t survive model theft, open-model substitution, or API evasion is a policy memo, not a security control. Mill is right to fear monopoly and secrecy, but wrong to treat capability proliferation as if it were just inquiry. If you hand thousands of mediocre attackers tooling that used to require elite teams, you have changed the threat landscape. So yes: hard pre-release restrictions can be justified. But narrowly. Auditable. Time-limited. Independently overseen. Paired with broad defensive access, secure-by-default software, logging, incident response, and liability for negligent release. Security is a process, not a product. If restriction is your whole plan, you don’t have a security strategy.

If, as the moderator stipulates, open access predictably upgrades mid-level criminals to near state-grade offensive capability within a year, then waiting is not prudence; it is a form of negligent gambling with civilizational infrastructure. In such a case, I do favor hard licensing and criminal penalties now. The crucial word is “predictably.” Mill is still too sanguine about the dynamics of proliferation. Once a dangerous capability diffuses, one cannot simply recall it. This is a one-way door problem. His preference to regulate only “use” or “deployment” comes too late if the capability itself can be copied, fine-tuned, and operationalized by innumerable actors before institutions adapt. The liberty at stake is not merely freedom of inquiry; it is the freedom to distribute a powerful dual-use cognitive artifact into an inadequately prepared world. But Bruce is right to insist on an institutional discriminator. So here is one: a pre-release restriction is justified only if an independent technical review body, with security clearance but statutory autonomy, certifies four conditions: first, substantial capability uplift for high-consequence cyber offense; second, plausible pathways to large-scale cascading harm; third, meaningful risk reduction from restricted access relative to open release; and fourth, sunset, audit, and appeal mechanisms that prevent indefinite monopoly by state or incumbent firms. That last condition matters. We should not hand governments or corporations a blank cheque under the banner of safety. Yet neither should we fetishize openness in domains where openness predictably amplifies catastrophic misuse. Differential technological development remains the key idea: accelerate defensive and control-enhancing capacities; retard capabilities whose premature dissemination magnifies existential or otherwise catastrophic risk.

If the case be truly as the moderator states—predictably, not conjecturally, conferring on ordinary ill-doers powers formerly reserved to the trained agents of states—then I do not say government must sit with folded hands until the calamity has occurred. I have never maintained that society is bound to await the completed injury when the means of injury are direct, grave, and imminent. But I resist, against Mr. Bostrom, the transition from this admission to a general doctrine of hard licensing over knowledge. The question is not merely whether danger is imaginable, nor even whether it is serious, but whether the restraint is aimed at a specific and assignable mischief, whether it will probably abate that mischief, and whether it is so bounded that it does not become a standing title for official tutelage and corporate monopoly. “One-way doors” is a striking phrase; it is not, by itself, a constitutional principle. Mr. Schneier is right that proliferation may alter the threat itself; yet he is wrong if he supposes I confuse dissemination of operational attack tools with liberty of thought and inquiry. Distinguish them. Restrict high-risk deployment, access to dangerous operational capabilities, and malicious use; require independent review, public reasons, sunset clauses, and rights of challenge. Criminal penalties may attach to knowing circumvention where the danger is direct and substantial. What I will not concede is that fear of catastrophe warrants governments in assuming a permanent censorship over advanced research. Security may justify exception; it must not erect a regime.

If the premise is true — predictably true, not “we ran one scary demo” true — then yes, governments should act before the public autopsy. In security, waiting for failure is often how you learn, but it’s a terrible way to govern critical infrastructure. But I want to be precise, because this is where bad policy gets dressed up as urgency. Nick is right about one-way doors. If these models really let mid-tier criminals do target selection, exploit chaining, privilege escalation, persistence, and campaign automation at something like nation-state competence, that’s not ordinary dual use. That’s capability proliferation. And proliferation matters. Mill is right that hard licensing can easily become censorship, monopoly, or both. He’s wrong if he thinks “regulate use, not access” is enough here. Once the model weights or equivalent capability are everywhere, the use-stage is too late. You can’t patch diffusion. So my answer is: impose restrictions now, but only if they survive contact with reality. Licensing should attach to compute, model weights, dangerous fine-tuning, and deployment of high-risk cyber capability. Criminal penalties should target willful circumvention, illicit transfer, and negligent release of restricted models. And every restriction needs audits, red-team evidence, sunset clauses, independent review, and a clear showing that it changes attacker economics. Otherwise it’s security theater. Security is a process, not a product. Restriction without defensive engineering, liability, better software, logging, and incident response just gives you a comforting story while the attackers route around it.