TopoBench  /  RL task packs

Tasks graded by recomputing the answer.

Every TopoBench task is generated from a seed, and its verifier works the answer out again in exact arithmetic. No model judge, no stored key to leak.

How a task is graded

1. Generate. An instance is built deterministically from a seed. With a private seed, it is in no public set.
2. Answer. The agent returns one strict JSON object. Extra text, floats and duplicate keys are rejected.
3. Recompute. The verifier checks the instance hash, solves the task again exactly and compares.

from topobench import betti_task as bt

inst = bt.generate(seed=42, difficulty=2)
h = bt.instance_hash(inst)        # kept where the agent cannot write
print(bt.instruction(inst))       # the text the agent sees

bt.verify(inst, '{"betti": [1, 2, 0]}', expected_hash=h)

Families

Topology and graphs

Betti numbers, minimum cycle bases, Forman-Ricci curvature, graph isomorphism.

Persistent homology

Barcodes of filtered complexes over the rationals.

Search and induction

Job sequencing, program induction, cage Latin puzzles.

Finance and back-office

Invoice reconciliation, expense audit, timesheet pay, inventory, month-end close.

Statistics

Multiple-testing corrections and calibration metrics in exact rational arithmetic.

Coding in a sandbox

Bugfix and feature tasks on small generated Python projects.

Back-office CLI suite

Operating a mock CRM, help desk and expense app through a command line.

Interactive sessions

Multi-step tasks against a live service: an oracle search under a query budget, a flaky-sync ledger task, and a clarify-then-report task. The clarify task has not been run on any model.

Every family has graded difficulty levels, ten for the single-answer families. The top levels change the rules or the instance distribution so the lower-level strategy fails by construction. What has been measured on real models is on the Difficulty page.

Grading guarantees

• Exact Fraction arithmetic, with no floating-point tolerance to game.
• A hash check fails closed if the task was edited to match a wrong answer.
• Coarse result codes, so the verifier output does not leak the answer.
• Graded scores beside pass/fail on the families that define one.
• Signed episode files: for released measurements, each episode is signed, so an edited or spliced results file is refused when graded.

Where this stands

• Every episode type can run inside a Docker container. This was checked on one Mac with scripted agents, not yet with a real model.
• Exports for two outside evaluation frameworks were run through those frameworks' own checking tools using a reference solver and deliberately wrong answers. No model was used. Details on the Method page.
• A scripted red-team of the interactive families found 6 issues. All 6 were fixed the same day.
• Difficulty is still pending: no pass rates are published. No blind human check and no expert audit have been done.