Weight Space Programming
The Dawn of a New AI Paradigm: Weight-Space Programming
Why do AI research like a human when you’re a machine?
Follow
When I first learned about neuralese, a name sometimes given to the unintelligible language AI systems use to communicate with one another when humans are not watching, one rather imaginative question occurred to me: why couldn’t machines put that alien fluency to work on the weights of neural networks directly? Or put differently: Why do AI research like a human — writing code and tuning hyperparameters and so on — when you’re a machine? Your body is made of numbers, and the universe is your playground: show me some magic.
The answer why they couldn’t, I realize now, was that the Bitter Lesson hadn’t reached this particular corner of the business yet. This is the answer for everything, by the way: if AI can’t do something it’s only because the implications of the Bitter Lesson don’t extend that far yet. That’s a slight exaggeration but not so outlandish anymore. This lesson has an embittering habit of being right.
In the most succinct formulation I’m capable of, the Bitter Lesson says that “handmade” methods always, eventually lose against scalable methods when it comes to AI. For instance, instead of introducing handcrafted domain knowledge like “good chess moves” or “good writing practice” — which might win in the short term — it’s better to just let the models learn and search for the best ones themselves. Let the machines do their job without imposing our own assumptions on them. It’s better to devote your efforts to exploiting the advantages of powerful computation.
In 2019, when Richard Sutton enunciated it, he was making an empirical observation with a few examples to draw from (games like chess and Go or visual recognition). Today, every single neural network — large language models are particularly big and particularly verbose neural networks — is an example of the lesson. Use more data, more compute, make the models big and scale up your algorithms and success is mostly guaranteed to follow.
A theory ought, out of decency, to become more complicated the more of the world it explains. Scale maximalism has shown no such manners. When GPT-2 learned to put together strings of symbols that made sense the main reason was scale. When GPT-4 learned to solve math problems and write competitive-grade code the main reason was, again, scale. When GPT-6 learned to handle Millennium Prize Problems the main reason was, yet again, just scale.
Naively, I was expecting the Bitter Lesson not to be so bitter; not to win every battle. It’s hard to watch so much human ingenuity surrender to so simple a principle: AI progress doesn’t quite require much of a human signature. But that’s how it is. And so I see no reason why AI research should be granted an exemption. Frontier AI models will eventually reach into the machinery of intelligence itself and patiently and carefully will weave its numerical DNA to their will, starting off an unprecedented era of recursive self-improvement without so much as a human inside the loop.
This is no longer fiction.
A team from MIT just published a study that suggests it’s possible.
They call this emerging paradigm “Weight-Space Programming.” Through eight experiments, they discovered that frontier models such as GPT-6 Astra can read, write, interpret, and edit neural network weights directly, like a human does ordinary code. Agents of evolutionary design, we might call them. There are caveats with this paper — the results are far from definitive and some have criticized the framing as exaggerated — so let’s see what’s going on.
II.
But first, let me talk about the premise because it requires a bit of background.
Neural networks learn by adjusting their weights, enormous collections of numerical parameters that determine how they behave. Ordinarily, these adjustments are made through optimization algorithms. That is, you set the gears of mathematics in motion and through clever sleight of hand — it’s quite clever, that’s why the best researchers make \$1M+ total compensation — the weights approximate the ideal values so that the network learns the intended behavior.
That’s what we call training.
These researchers wondered whether a sufficiently intelligent large language model (LLM) like GPT-6 could bypass the training process altogether and decide what the weight values ought to be.
To appreciate the ambition, imagine sixteen years of schooling, examinations, lectures, and sleepless nights reduced to a brief appointment with a neurosurgeon. He opens your skull, makes a few adjustments to the wiring — a pyramidal neuron here, some synapses over there — and sends you home knowing everything you would have otherwise normally learned throughout your youth.
You may remember the same fantasy from The Matrix: Neo learns kung fu through a socket in the back of his head, while Trinity acquires the ability to pilot a helicopter in seconds. Except that here the proposition is stranger still: rather than downloading a ready-made program, another intelligence must figure out which connections to alter.
The researchers appointed GPT-6 Astra for the task, but I presume the results, insofar as they are replicable at all, will be easily reproduced with similarly powerful models like Opus 5.5. (As I was finishing this article, someone tried to reproduce the experiments and, sure enough, the answer is yes, Opus 5.5 can. He also showed weaker models are unable to do it, but then a different attempt at replication suggested the opposite: even weaker models can. Someone else has even made a new benchmark that ranks the models more granularly. Expect more news soon.)
Press enter or click to view image in full size
“Weight-space programming effectiveness across model generations.” Source
Press enter or click to view image in full size
Writing Weights Index. Source
It’s quite remarkable that an LLM, a general-purpose AI trained to predict the next token and reinforced to reason in natural language, can take a glance at those weights — the figures are in the dozens of thousands, not millions, which is a significant but modest size — and natively figure out what they do or how to tweak them so that their behavior changes in some particular way.
This could be, in a way, an evolution of the Bitter Lesson: it’s not that computational methods surpass handcrafted ones but that the output of computational methods — frontier AI models — surpass computational methods themselves! Optimization algorithms might succumb to the sheer power of a greater intelligence just looking. In another sense, funnier still, it could be a return to the traditional way of doing AI: first humans used handmade tricks and then humans discovered the power of letting computers do the tricks. Then those computers created a “new species,” AIs, which are now doing their own handmade tricks. So what comes next? You guessed it: AIs figuring out that they can leverage computational tricks, which will put AI research itself two degrees removed from human involvement or understanding: we won’t either craft the tricks not govern the computers.
But let’s not get ahead of ourselves and let’s take a look at what the MIT researchers found. And remember: the wise thing to do is to not miss the trajectory for the points.
**
III.
I’m not going to go through all eight experiments but just the ones I think illustrate best the idea.
In one experiment, the researchers gave Astra, as a text prompt, the weights of a small network trained to play some simple videogames. Naturally, the weights are already trained but to the naked eye they’re just random numbers all the same. Astra correctly identified which game each network had learned to play without watching it play.
Press enter or click to view image in full size
It’s just pattern matching, right? These things are trained precisely to do this kind of task. No, this is nuts. Even if this example is a “minimum viable research,” it hints at emergent capabilities that might have implications for training future AI models and for understanding existing ones. As far as I know — and, please, correct me if I’m wrong — this is the most relevant research avenue that tackles both the frontier of capabilities and safety at once.
Then they reversed the experiment.
Instead of asking Astra to read an existing network, they asked it to write one. Given the rules of three videogames and the architecture of an empty neural network (that is, the skeleton is there but not the substance), Astra generated the weights from scratch. Its creations outperformed networks trained using conventional reinforcement learning for 100 million steps, in all three games, without training, gradient descent or trial-and-error.
Press enter or click to view image in full size
Artists in the audience will know the distance between the two assignments: how much harder is it to take a blank canvas or page and turn it into a masterpiece than just read Moby-Dick or go see Las Meninas and inspect their many details.
The fact that an AI can conjure up a neural network on the fly, when the rest of the field has spent decades coaxing millions of numbers down the slopes of a vast mathematical landscape one infinitesimal adjustment after another — a method called gradient descent — is like witnessing the advent of a class of magicians in a world of Muggles.
Another experiment, perhaps the most challenging, involved a language model.
They asked Astra to improve Qwen3.5–0.8B’s mathematical performance by editing its weights (it’s a tiny model by today’s standards, I don’t think this would replicate at all, at least yet, in genuinely powerful models). It managed to raise the model’s MATH-500 score from 7.9% to 48.1% by changing just 0.016% of its parameters.
Press enter or click to view image in full size
But there’s a catch.
Qwen was prone to getting trapped in repetitive reasoning loops, sometimes exhausting its entire output allowance before producing an answer. Astra identified the problem and corrected the weights responsible. It hadn’t given Qwen any new mathematical abilities but had instead removed an obstacle preventing Qwen from exercising the abilities it already possessed using standard mechanistic interpretability techniques.
When the researchers tried the same approach on another language model, one without this peculiar problem, Astra failed to produce a meaningful improvement. That’s why I said not to get ahead of ourselves: the approach is genuinely interesting but, for now, the evidence is limited to frontier models being able to read certain neural structures, construct small networks capable of useful behavior, and make targeted repairs to larger ones. Whether they can create genuinely new capabilities inside existing LLMs remains an open question.
Now that you know what was actually tested and what the researchers found, let me say something. This is a genuinely interesting approach with, at best, fairly limited findings. It’s a perfect example of how easily an intriguing result in AI can be hyped into a revolution. Don’t fall for that. Wait for others to weigh in, for independent replication, for the less exciting explanations that tend to emerge once the viral threads have done the rounds, etc.
But if you want something truly explosive, then we’ll have to venture beyond the evidence and into speculation. So let’s — that’s my favorite part of this job.
IV.
Suppose that they do.
Suppose that GPT-10 could examine the weights of GPT-9, identify the circuitry responsible for whatever behavior, and alter it without retraining the network. Suppose it could transplant capabilities between models, remove undesirable quirks, or introduce new perks with surgical precision, without removing other capabilities (that’s known as catastrophic forgetting and it happens). Note that I’m not making a crazy assumption. I’m just extending the current results in line with what the Bitter Lesson would predict: scale up and the models will be able to do this.
That’s why I brought the Lesson up in the first place and why I insist not to look at the point but rather at the trajectory: what’s new here is not so much tweaking weights directly but the fact that it’s a general-purpose LLM that’s doing it. The implication is that the next generation will do it better.
The first-order effects would go well beyond cheap training or a greater understanding of the models inside out. Actually, to prevent underselling the consequences, I will say that this could radically change the field as much as deep learning did when it was combined with powerful parallel processors (GPUs) in the early 2000s. Much of the expensive, arcane art of persuading billions of numbers to arrange themselves into something intelligent could give way to deliberate engineering
But let’s go even further.
If an AI could perform these operations on a model more capable than itself — if GPT-10 could improve not just GPT-9 but even GPT-11 — then it would be a first in history: an intelligence would be able to design a greater intelligence than itself. (Before you interrupt my delusion, let me say that although Astra itself is smarter at some specific things than all humans, I don’t consider it at all smarter overall than essentially any complete human.) If that happened then recursive self-improvement would be immediately real and the consequences, unfathomable.
There’s another reason why this is a big deal.
Perhaps one of the great misconceptions about ChatGPT and its kin is that somebody actually designed them, in the ordinary sense of the word. LLMs are trained using enormous quantities of computing power and data, but the process has far more in common with a caretaker tending a garden than with an engineer building a car. The AI labs don’t specify the billions of numerical relationships that emerge within the models; they merely watch them blossom.
Not even the best AI researchers, know how to map ChatGPT’s weights onto its behavior. Its size makes it incomprehensible, its stochastic nature unpredictable, and its opacity impenetrable. Hence the term black box. Hence the notion that they are grown, not designed. Hence the enormous interest in advancing the field of mechanistic interpretability.
Which is why this discovery is so astonishing, and so unsettling at the same time: because on the one hand, it suggests that, before long, AIs might be designed rather than grown; and on the other, it raises the possibility that we, flesh-and-blood humans, apparently inferior at one more thing now, won’t be the ones doing the designing.