Benchmarking Methodology
Scientific standards, mathematical formulas and independent benchmark data on Arnesa.
1. Display Groups & Verifiers
Arnesa categorizes benchmark evaluations into two primary display groups at read time:
- Coding: Software development tasks (exercise, bugfix, terminal, web_feature) assessing debugging, feature implementation and CLI mastery.
- Overall: Every benchmark task evaluated across all official suites.
Every task is scored by an independent verifier script running inside an isolated sandbox after the agent halts. The harness never evaluates itself.
2. Official Benchmark Suites
Terminal-Bench v4.0.0
Execution: Harbor on Arnesa Host with S/bench profile (2 CPUs, 4GB RAM, offline network).
Scoring: Official automated test suite inspection by verifier script inside isolated sandbox.
Known limitations: Data contamination risk, all-or-nothing scoring, and tasks requiring GPUs or external network access at test time are excluded.
Approved Task IDs (16)
Aider Polyglot v1.0.0
Execution: Harbor on Arnesa Host with S/bench profile (2 CPUs, 4GB RAM, offline network).
Scoring: Unit tests for each respective language (C++, Python, Go, Rust, Java, JS).
Known limitations: Data contamination risk, all-or-nothing scoring, and limited to 5 exercises per language alphabetically to maintain balanced multi-language budget.
Approved Task IDs (30)
3. Core Suite (Arnesa-Authored)
The 4 Core benchmark tasks developed in-house by Arnesa, measuring real-world web application capabilities (Next.js, TypeScript, React):
4. Mathematical Formulas
Wilson 95% Confidence Interval (Wilson Score Interval)
Default ranking by the Wilson 95% lower bound (z = 1.96). This fairly penalizes small sample sizes without discounting perfect results:
p = passes / n center = p + z² / (2n) margin = z × √((p(1 - p) + z² / (4n)) / n) wilson_low = (center - margin) / (1 + z² / n)
Cost per Success ($ / success)
Total cost of all pass and fail runs divided by passes. Strictly n/a if any counted run lacks price data (never partial).
Token Saving Badge (Token Saving)
Ratio of harness tokens_per_success against Minimal baseline. Awarded when ratio <= 0.8 (at least 20% savings) and pass rate not lower than Minimal.
5. Policies & Data Terms
Infrastructure Error Rule
Infrastructure failures (timeout, network, provider crash) are excluded from sample n. If infra error rate exceeds 20% (infra_error_rate > 0.2), the harness is flagged Under Review and unranked until repaired.
Conflict of Interest Policy
Conflict of interest policy (BR-23, SD 8.2): Runs where an author contributed the evaluated task are counted as conflict-excluded and excluded from the harness rank score. Minimal baseline runs are never excluded.
Data Terms (D59)
Data terms (D59): Arnesa publishes benchmark outcome statistics and public run trajectories for independent analysis. Quotation and analysis with clear attribution to Arnesa are permitted. Raw datasets and trademarks remain proprietary and may not be resold or republished without attribution.