Breviv0.28

How it works

Speed by build path

This page is built from docs/PATH_SPLIT.md in the repository; its data is bench/pathsplit.

View as Markdown · source

Brevi builds a Python file one of three ways: the typed build (Rust with native types, the fast one), the dynamic build (a native binary with Python's object model, for programs whose values change type), or pybridge (the program runs through an embedded CPython). Which one a program gets decides how fast it runs, so a single speed figure says little without the share of programs that get it. This page measures both on one real workload: Python that models wrote to finish benchmark tasks.

The short answer#

On 587 distinct agent-written programs, built with Brevi from the v0.28.2 release tag (commit 66d196da):

TypedDynamicPybridgeRefused
Programs that took the path366 of 587 (62.4%)221 of 587 (37.6%)00
Printed what CPython printed, every test365221--
Stopped with the typed build's checked error10--
Wrong answers00--
Median time per test, CPython11.7 ms17.3 ms--
Median time per test, native binary1.1 ms2.3 ms--
Whole-run speed-up over CPython, median10.8x7.4x--
Range8.5x to 18.6x3.1x to 9.0x--

Read the speed rows as start-up, not compute. These programs get small test inputs: CPython needs about 10 to 12 ms just to start and print one line on this machine, so most of its 11.7 ms and 17.3 ms per test is the interpreter starting and importing modules. A native binary starts in about 1 ms (typed) or 2 ms (dynamic). The speed-ups above are what an agent sees when it runs many short programs; they are not how fast either build computes.

Compute speed by path comes from the released numbers (NUMBERS.md, bench/compare's 12 compute programs, measured on Brevi 0.27.1 and not yet re-measured on v0.28.x): typed median 9.3x, dynamic median 0.85x against CPython, and only a small share of programs take the typed build (2.6% of core conformance matches). A long-running program on the dynamic build runs at about CPython's speed, slower on numeric loops; only the typed build is several times faster.

The one program that stopped (n1-sonnet-python/stack_vm.py) masks numbers with (1 << 64) - 1, which does not fit the typed build's 64-bit integers. Its binary printed nothing, gave the error the typed build stopped: an int passed 64 bits and exited 1, so it is not a wrong answer: brevi run re-runs such a program on the dynamic build, and BREVI_DYN=1 builds it that way.

The typed share falls as programs grow#

Program size (lines of Python, all files)Took the typed build
Under 40205 of 250 (82.0%)
40 to 79105 of 132 (79.5%)
80 to 14953 of 133 (39.8%)
150 or more3 of 72 (4.2%)

The median program here is 48 lines. Short, single-purpose task solutions mostly build typed; programs of a few hundred lines, and 30 of the 32 multi-file programs, mostly do not. The same pattern shows in Brevi's other samples, so the typed share depends heavily on what is measured:

So the typed path covers most small agent scripts but only a small fraction of larger programs and of CPython's test suite.

Why programs leave the typed build#

The build prints its first reason. Grouped, over the 221 dynamic programs:

ProgramsReason
57Two names share one list and one of them is changed while the other is still used (the typed build would change a copy)
40A loop, comprehension, list(), str() or % over a set: CPython walks it in hash order, which the typed build would not reproduce
30is between values other than None, True, False and Ellipsis needs object identity
29The typed build's generated Rust did not compile
19A value changes type
19A library or method the typed build does not have (datetime, decimal, re.findall with groups, ...)
2An int past 64 bits
25Other: a type the build could not work out, a name defined in a way it does not follow, a return out of try/finally, nonlocal on a list, two parse gaps

Apart from the compile failures, each is the typed build declining rather than guessing: when it cannot show it prints what CPython prints, the program goes to the dynamic build. The 29 Rust compile failures are compiler gaps, caught at build time and sent to the dynamic build too; they are the cheapest typed share to win back.

How it was measured#

Rerun it with a brevi built from a release tag (on Linux, as root, for the box):

git checkout v0.28.2 && cargo build --release && mkdir -p target/rel && cp target/release/brevi target/rel/
git checkout main
BREVI=target/rel/brevi python3 bench/pathsplit.py target/pathsplit.json
python3 bench/pathsplit.py --report target/pathsplit.json

Limits#