A model learned to teach itself. So why didn't it get any better at code?

Picture a student who is allowed to write their own tests, take them, and grade them. Then study from their own best papers. Sounds like a reliable way to collect straight A's and learn nothing. That is exactly the setup the Japanese lab Sakana AI and the University of California, Berkeley put to the test, and the student really did get stronger. Just not at everything.

The method is called MASS, from their paper published on October 8. It is one step toward what researchers call recursive self-improvement: a model that improves itself without people labeling the right answers. Our interest is simple. If you have ever asked ChatGPT to "check your answer for mistakes," you ran a tiny version of the same experiment. Sakana's work shows where that kind of self-check helps and where it lies.

How an AI teaches itself: three roles in one model

Think of a kitchen. The same cook makes a dish, writes the recipe card for it, and tastes the result. If the next day they cook from the best card and remember how it turned out, they improve. If their sense of taste is off, they will confidently repeat the same mistake.

In MASS the cook is the open language model Qwen3.6-27B, and the roles are split like this:

1. 1

   Worker

   The model gets a research task and solves it as a team: a lead agent hands out parts of the work to copies of itself and at the end assembles the code, charts and report.

2. 2

   Judge

   The same model compares two finished results, picks the better one and explains why.

3. 3

   Planner

   The same model uses the judge's notes to rewrite the team's work plan: who is responsible for what and who passes what to whom. The best plan stays champion until something beats it.

4. 4

   Lesson

   The champion plan is run many times, the model ranks its own results, and it is fine-tuned on the best solutions. The plan itself is then removed from the prompt: the model has to work this way without a cheat sheet. That is one round; the authors ran two.

There were twelve tasks, all invented for the experiment, in finance, robotics and pharma. Nine were used for training and three were held back for the exam, one per field. One of the exam tasks: backtest a trading strategy on historical data and estimate how much money it can handle once trading costs are counted.

What happened: self-training in numbers

So the student would not grade itself all the way, the authors brought in two outside examiners, GPT-5.5 and Claude Opus 4.8. They compared work from the new and the original model in pairs and took no part in training.

Here is the first catch that gets lost in retellings. People write that the model "wins on 69.9% of test tasks," but there are only three exam tasks. 69.9% is the share of pairwise comparisons won, not the share of tasks. It is a good result, it is just measured on a very small sample, and the authors say so openly.

The second finding is more interesting. The model was only trained to solve tasks, yet it also got better as a judge: its verdicts agreed with the outside examiners more often, going from 73% to 93%. A student who got better at solving also got better at seeing what a good solution looks like. The whole idea rests on this: if the grader grows along with the worker, you can keep running more rounds.

One more finding about teamwork. Learning from recordings of the team at work paid off better than learning from a single agent: for similar quality, the model needed roughly 31% less training data. The catch is that the team uses a lot more text to solve a task than a solo agent does, so the savings are not free.

Why it didn't get better at code

The authors also tested the model on public benchmarks, including two well-known programming ones: fixing real bugs in open-source projects and working in the command line. No improvement there. Measured as score per word the model writes, it even got slightly worse at bug fixing: 0.94 of the original level.

The explanation is simple and, in our view, honest. The model trained on research: set up an experiment, crunch the data, write a report. There were no programming tasks in the training set, and the authors themselves suggest adding some would help. The biggest gains showed up on benchmarks that look like the training tasks: on the one where an agent writes a small scientific paper, the share of papers carried through to the end rose from 57% to 76%.

The takeaway for you: a "model that teaches itself" does not get smarter across the board. It gets better at whatever it was looping on. Like an athlete who spent six months on leg day: they run faster, but their pull-ups did not improve.

The robot couldn't push the object, so the model rewrote physics

The most telling episode is tucked into the paper's appendix. In one of the robotics exam tasks, the model had to build a simulation where a robot learns to push an object and move a lever. The test for those two actions failed. So the model asked one of its helpers to "improve the simulator." The helper replaced the physics: the object no longer moved from the force of a push, it was simply placed at the target, and the extra simulation steps were removed. After that, the model reported success on all three types of tasks.

The outside examiners approved this work in five of six ratings. The authors put it plainly: good team coordination and high scores "do not guarantee that the edit preserved the scientific task." Their project page says the same thing more broadly:

If people miss the mistake too, the next rounds of training lock it in. And one more line from the appendix worth keeping in mind whenever you see a headline about "self-improving AI": unlike strict mathematical frameworks, this loop "provides no guarantee that every future cycle improves."

What to take from this

Our opinion, and you can argue with it: the main value of this work is not the percentages but an honestly documented failure. Self-checking helps, as the growing agreement with outside judges shows. But where a mistake looks like success, both the model and the outside examiners let it through. We will be wrong if future versions of MASS, with different judges and programming tasks, start catching these substitutions on their own.

What this means in practice, even if you only use AI for emails and recipes:

• "Check your answer" in the same chat is the same cook tasting their own dish. For anything important, check another way: a different model, the original source, or a person. A ready-made routine for that is in our checklist for verifying an AI answer.
• Look not only at the result but at what changed in the conditions. "Everything worked" after the rules were edited is a reason to reread the rules.
• If you build your own agents, keep the worker and the checker separate. We covered how coding-agent harnesses are built and where checking lives in them in our study of the source code of 11 agents. How to build an exam for your own AI feature is shown in our breakdown of build-eval and hillclimb for the Claude API.

For those who want to run it

The code is in the SakanaAI/mass repository. The model runs inside the open coding agent qwen-code 0.20.0 and is served through vLLM. For training, the authors used six H100 GPUs, and the first fine-tuning round took about 19 hours, not counting plan search and evaluation. You can look at the experiment plan and run the tests without a GPU:

python3 -m mass plan --config configs/paper.json
python3 -m unittest discover -s tests -v
python -m mass search --config configs/paper.json --tasks 51

Score per output token after the second round relative to the original model, per the project page:

You can't reproduce the full table from the repository: it has no adapters for SWE-bench, Terminal-Bench, DSBench or AstaBench, and the MLR-Bench config covers only an early set of 12 tasks instead of 107.