Experiment 1 · nanochat
7 rounds of rewrites. Same 20 minutes. A model that guesses better.
nanochat is an open project for training small language models. The rules are fixed: one A800 GPU, 20 minutes per training run. Raven (powered by DeepSeek v4.1-flash with the ai4ai plugin) read each round’s results, edited the training script train.py itself and launched new runs, for 7 rounds.
val_bpb, lower is better
The official train.py starts at 0.9956. The round-7 recipe reaches 0.9381 (mean of 3 runs with different random seeds, ±0.0007): 5.8% lower.
train.py starting point. Hover a dot for its values.Not done yet: the experiment stopped because the GPU budget ran out, not because the gains did.
Each of the last two rounds (6 and 7) improved by far more than the run-to-run noise (0.0013), and the curve is still heading down. Over 7 rounds: 172 training runs, not one crash, 3586 of the 3600 budgeted GPU minutes used.
What is val_bpb?
It measures how surprised the model is by text it has never seen. The model predicts the text one piece at a time, and the uncertainty of those predictions is converted into bits per byte of text. The lower the number, the better the model guesses.
The 7 changes the final recipe makes to the official code
- Half the data per step (batch 219 → 218), so 20 minutes fit 3055 steps instead of 1540
- A new learnable lookup table for pairs of neighbouring tokens, 100× the vocabulary size (819200 rows)
- The table’s output is added to the model right after layer 0
- Learning rate
MATRIX_LRlowered from 0.04 to 0.02 - Short attention window cut from 1024 to 256
- A wider model:
ASPECT_RATIO64 → 80 (width 512 → 640) - Evaluation batch fixed at 128
Best recipe in each round
| Round | Recipe | val_bpb | Seeds | GPU min |
|---|
Notes: the final recipe has only 3 seeds so far; Raven’s own report recommends 7–8. The wider model is about 30% slower per step, so 20 minutes fit 2363 steps instead of 3055, and it still scores lower. That is the net effect under a fixed time budget, as the rules intend. A “real improvement” means beating the previous round’s best with Welch |t| > 3 and |Δ| > 0.002. None of round 1’s 9 candidates passed; every round from round 2 on set a new best.
Experiment 2 · CFD dam-break simulation
Three crashes chasing speed. Then it backed off and shrank the error, round by round.
CFD (computational fluid dynamics) means simulating how liquids and gases flow. This run used the open-source OpenFOAM to simulate a 2D dam break: a column of water collapses and slams into the far wall. Same case, 81648 cells, 7 rounds (0–6). Each cell records what fraction of it is water (called alpha), which must stay between 0 and 1. A negative value, or one above 1, is an unphysical error: the cell is “out of bounds”.
Round 0Safe settings, ran to the end
Round 1Faster, crashed at t=0.4
Round 2Faster still, crashed at t=0.5
Round 3Faster still, crashed at t=0.65
Round 4Back to safe, ran to the end
Round 5Smaller error, blurrier surface
Round 6Smallest error
First, chasing speed: bigger and bigger steps, three crashes
Rounds 1–3 made each step forward in time larger and larger (the maxCo setting, raised from 1 to 20, 100 and 500). Out-of-bounds cells jumped from 0 to 241, 3877 and 1241, and none of the three reached the end.
Then, backing off: the error shrinks every round, but isn’t zero yet
Round 4 set maxCo back to 1: it ran to the end and conserved the water volume. Rounds 5 and 6 refined how the water surface is computed, and the out-of-bounds size kept falling. The criterion demands the water fraction stay strictly within 0 to 1, which floating-point math can’t guarantee: rounds 4–6 all missed by just 1e-7 to 1e-10, so they are still marked as not passing.
What changed in each round
| Round | Change | Result |
|---|
Note: out-of-bounds counts and sizes were measured by the rendering script; Raven’s own report measured at a different place, so this page uses the rendering script’s numbers throughout. Round 4 is recorded as an alpha minimum of −1.97e-07; the chart plots its size, 1.97e-07.
Experiment 3 · FEA cantilever beam
Halving the gap like a guessing game: 8 rounds to close in on where the software gives up
This is the software’s computing limit, not a force the beam could actually carry. FEA (finite element analysis) splits a structure into many small pieces to work out the forces on it. In this run (2026-08-17), the open-source CalculiX loaded a beam fixed at one end. The bigger the load, the harder it is to compute; past a certain point the software fails to converge, meaning it keeps iterating without reaching an answer. Raven hunted for that point by bisection: if it computes, push harder; if not, back off; always try the midpoint.
Big steps first, then halves
Round 0 at 2000 kN failed; round 1 dropped to 1800 kN and computed. From then on, each round tried the midpoint between the largest load that worked and the smallest that didn’t.
Pinned between two loads
1962.5 kN still computes; 1965.625 kN doesn’t. The run was logged as “bisection converged” and finished normally, with 107 events recorded along the way.
The beam itself could never take this
From its material and cross-section, the beam’s theoretical plastic limit is only 62.5 kN, and every load in all 8 rounds is far beyond it. About 1962.5 kN is simply the largest load the software can still finish computing.
Load and result in each round
| Round | Load (kN) | Result | Run time |
|---|
Experiment 4 · AI4AI cost comparison
Raven on a cheap model: a fraction of the cost, still at the front
A separate experiment with different budget rules from nanochat above, so the numbers aren’t comparable. The task: use nanochat to prepare a 50M pretrained model, scored by BPB (how badly it predicts text; lower is better; the A800 baseline is 1.108). Raven or Claude Code, each paired with a model, competed under two budgets.
- Raven · V4.1 (cheap model)
- Raven · other models
- Claude Code · Opus 5 (for comparison)
5 hours, no spending cap
The same 1.037 league, 40× cheaper. V4.1 cost $0.97; Opus 5 cost $39.41. On the same Opus 5, Raven scored 1.038 and Claude Code 1.053.
$50, no time limit
$6.12 spent, best score of all: 0.965. The three pricier setups used up the budget; cheap V4.1 could run for 80+ hours and explore more.
Full data for both budgets
| Setup · model | Time | Tokens | Cost | BPB |
|---|
| Setup · model | Time | Tokens | Cost | BPB |
|---|
A harness doesn’t make the model smarter. It makes the model work reliably.
Don’t trust what the model says about itself; trust only evidence you can check.
How Raven does it
- Multi-agent orchestration
- Agents hand over artefacts by contract instead of negotiating opinions; writing code and running experiments are kept apart so measurements stay clean; shared memory supports long runs and saves tokens.
- Research
- Strategy adapts to budget, how much has been retrieved and whether it is enough; difficulty is judged from retrieved evidence, not assumed up front.
- Code
- Beyond implementation: independent acceptance checks and tests; objective facts are read by tools, not taken from the model’s own account.
- Watch duty
- Submits jobs itself, schedules check-ins, decides to redo or stop; writes its own timers, heartbeats and session keep-alives.
- The ai4ai loop
- Propose directions (DR + breadth-first) → implement → check validity → measure → close out with evidence, or dig deeper depth-first.
- Idea seed bank
- Records which directions have and haven’t been tried, so Raven knows where things stand and can explore and combine new ones.