How it works

Benchmarks

View as Markdown

Brevi is measured two ways: what it costs an AI model to reach a working program, and how the built programs behave, against CPython and against the other Python compilers. All benchmarks, logs and scripts are in the repository's bench/ folder, so every number here can be rerun.

The release test#

The release test is the bar every release has to pass. Claude Sonnet and Claude Haiku each solve the 24 tasks of the rel suite twice per side:

• python: the model writes Python and checks it with CPython until the tests pass;
• pyb: the same prompt, and the same Python, checked with Brevi.

Each side's prompt cache is kept apart so neither pays the other's cache writes. The whole API bill is counted, retries included. Since 3 November every Brevi check also runs the same files with CPython (the "shadow check"), so a check that failed only because of Brevi is counted as a Brevi bug.

The bar: every task passes, Brevi's cost is within 10% of CPython's averaged over the runs for both models, and at least 90% of final programs build natively, with no CPython fallback.

Passed in every row. In r244a to r244e (r244e on the release commit, the rest on main just before it), five checks (in the trains and poker tasks) failed under Brevi where CPython passed the same files: the typed build printed a wrong answer or stopped with an error. Each model fixed its code and passed; the cases are saved and being fixed for the next release. The one-run checks after v0.24.0 are rerun after each change to the compiler that could move cost; the two fallbacks left in w156a were both json_query, whose json.JSONDecoder().raw_decode builds natively on main since then. The earlier runs e31 to e34 (2 November) also passed: 48 of 48 tasks, all 48 final programs native, cost 1.10x (Sonnet) and 0.95x (Haiku).

The bar is "no more expensive than Python", not "cheaper", on purpose: on ordinary tasks a model writes much the same Python either way, so the cost should come out level, and the speed and safety of the result come free. The test exists to catch Brevi making an agent work harder, by rejecting valid Python or answering differently.

Heavy tasks and the plain-solution line#

The speed set has four tasks with small inputs and tens of millions of steps under a 10-second limit: Game of Life up to 600x600, 0/1 knapsack, edit distance on 10,000-letter texts, and Collatz chains. A plain Python solution takes CPython 15 to 80 seconds, so a model writing for CPython has to hand-optimise.

With one extra line in the prompt telling the model its code is compiled and to write the plain solution (see For AI agents):

Every task passed on both sides, every Brevi program built natively, and no check failed because of Brevi. The saving is the model not hand-optimising: output tokens fell by about half for Sonnet and a quarter for Haiku. On the ordinary release-suite tasks the same line made no clear difference (runs v1a, v1b: Sonnet 1.07x, Haiku 1.00x, bar still met).

Brevi, Codon and Nuitka#

The same unchanged Python files were given to each compiler, and every built program was checked against CPython: stdout byte for byte and the exit status. Brevi is brevi build --release; Codon is codon build -release -exe; Nuitka is python -m nuitka --follow-imports.

Speed, best of 3 runs, in seconds:

Brevi's default build (what brevi run uses) is 1.7x to 40x faster than CPython on every program; --release is level with Codon overall, and Brevi builds programs Codon cannot. Nuitka matches CPython's behaviour like Brevi, but runs close to CPython's speed because it keeps CPython's object model. Measured 3 November 2026 on one 4-core Linux machine (Codon 0.20.6, Nuitka 4.3.4, CPython 3.13, rustc 1.97).

Rerunning#

cd bench
python3 e2e.py RUNa --lang pyb --model sonnet --set rel --jobs 4      # the release test (needs an Anthropic API key)
python3 e2e.py RUNa --lang python --model sonnet --set rel --jobs 4
python3 e2e.py --verdict "logs/e2e/RUN[a-d]-*.jsonl"
cd ..
python3 bench/compare/compare.py --corpus solutions --tools brevi,codon,nuitka --jobs 4
python3 bench/compare/compare.py --corpus speed --tools brevi,brevi-dev,codon,nuitka

bench/README.md has the full method and every run since the project began, including older token-count benchmarks of Brevi's own .bv language against Python and Rust. Browse it on GitHub.

The history, briefly#

Brevi began as a language of its own for models to write, measured by tokens per finished task against Python. It got close (within a few percent either way on most sets), but the measurements kept showing that models write Python more cheaply and more reliably than any new syntax, however short. So the goal moved: take the Python models already write, and make it fast and safe. The .bv language is still there for anyone who wants it.