Piper · Coding harness
Log in

Summary / The 27 components

11 Code execution

Running code under a boundary and returning observable evidence rather than trusted text.

Latest—not scored
Baseline— 
Live use—last 7 days

Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.

What is built now

exec.py: every harness run goes through a systemd user scope (task and memory caps), bubblewrap (read-only system, no home, no network, private /tmp, own pid namespace, cleared environment) and prlimit (CPU, file size), with output captured and truncated. run_test turns a pytest target into per-test verdicts and splits very long target lists that once overflowed the kernel's argument limit. SWE-bench runs go into the instance container through interpreter shims. Since 10-04 Docker itself is reached only through a root-owned broker that accepts benchmark images and harness mounts; before that, a scoped wrapper inside the account (09-26) did the same check.

What is missing

Failure normalization into classes (syntax, type, assertion, timeout, …). The diagnoses table has never had a row.

Reviewer Claude, 2026-10-04; close-out verdicts 2026-10-23

Reviewer's read: Passes check (10-23), after a sandbox fix

The strongest piece of infrastructure in the programme. The 10 escape probes in the hostile corpus run against this exact code, and they passed again on 10-04. Two real bugs were found and fixed in use: tests that only failed when the working directory was the host mirror path (the container now runs with cwd /testbed), and the argv overflow above. The boundary covers the harness's own runs. It does not cover the worker's shell (see 24).

Close-out, 10-23 (pre-registered check, Claude): the sandboxed-execution tests pass, after a fix found by this check. Passes check. The fix: since the 10-16 move to the new machine, the harness venv's python is a link to an interpreter outside the venv, and the sandbox did not mount it, so every sandboxed run of that python failed (6 tests in execution, test running and acceptance). The sandbox now mounts that interpreter tree read-only; nothing else in the home directory is visible. The boxed bench never used this path, so no score is affected.

How it is measured passes check (10-23, fixed)

Hostile corpus (escape probes) on every harness change; scorer pilot (gold resolves, empty patch fails) on every rig change.

Adoption rules shared by every arm: How the tests work.

Research & attempts

Score history

No scores yet. The first entry is the baseline for suite v1.

Experimental record

One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.

No interventions recorded yet.