Recursive self-improvement is the only capability worth a moratorium

The threshold problem

Every serious governance proposal now contains a capability threshold, and almost all of them are written in terms of evaluation scores. The reason is practical rather than principled: scores are legible to lawyers, they can be written into a statute, and they can be audited after the fact. The cost is that they measure the wrong thing.

A score describes what a system can do on a fixed task distribution at a fixed moment. The transition the assembly is concerned with is not a level of performance but a change in who is doing the improving. A model that writes a better training recipe for its successor has not necessarily moved on any published benchmark.

The question is not how well it scores. The question is what it is allowed to change.

What a threshold over modification would look like

Replace the score with a permission boundary. Define, for a given system, the set of artefacts it may alter: its own weights, its training data, its objective, its evaluation harness, the code that runs any of those. A threshold crossed is not a number exceeded but a boundary widened.

This has an auditable form. A laboratory can state which of those artefacts its systems are permitted to touch, and an outside party can check the claim against the infrastructure without seeing a single model weight.

What the assembly does not claim

Nobody in these sessions argues that a general moratorium is enforceable or that current systems are close to the boundary described here. The narrower claim is that the boundary is the right object to regulate, and that writing it down now is cheaper than retrofitting it later.

Published by Invariant. No author is named, by design. Invariant publishes views held by people who work on these systems and does not publish what their employers know.