Summary / The 27 components
11 Code execution
Running code under a boundary and returning observable evidence rather than trusted text.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
exec.py: every harness run goes through a systemd user scope (task and memory caps), bubblewrap (read-only system, no home, no network, private /tmp, own pid namespace, cleared environment) and prlimit (CPU, file size), with output captured and truncated. run_test turns a pytest target into per-test verdicts and splits very long target lists that once overflowed the kernel's argument limit. SWE-bench runs go into the instance container through interpreter shims. Since 10-04 Docker itself is reached only through a root-owned broker that accepts benchmark images and harness mounts; before that, a scoped wrapper inside the account (09-26) did the same check.
What is missing
Failure normalization into classes (syntax, type, assertion, timeout, …). The diagnoses table has never had a row.
Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23
Reviewer's read: Passes check (10-23), after a sandbox fix
The strongest piece of infrastructure in the programme. The 10 escape probes in the hostile corpus run against this exact code, and they passed again on 10-04. Two real bugs were found and fixed in use: tests that only failed when the working directory was the host mirror path (the container now runs with cwd /testbed), and the argv overflow above. The boundary covers the harness's own runs. It does not cover the worker's shell (see 24).
Close-out, 10-23 (pre-registered check, Claude): the sandboxed-execution tests pass, after a fix found by this check. Passes check. The fix: since the 10-16 move to the new machine, the harness venv's python is a link to an interpreter outside the venv, and the sandbox did not mount it, so every sandboxed run of that python failed (6 tests in execution, test running and acceptance). The sandbox now mounts that interpreter tree read-only; nothing else in the home directory is visible. The boxed bench never used this path, so no score is affected.
How it is measured passes check (10-23, fixed)
Hostile corpus (escape probes) on every harness change; scorer pilot (gold resolves, empty patch fails) on every rig change.
Adoption rules shared by every arm: How the tests work.
Research & attempts
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.