CLI reference

Every loop action is available from the ratchet command, and every command calls the same orchestrator functions the dashboard API calls. Nothing is CLI-only, and nothing in the dashboard is missing from the CLI.

Run commands with uv run from the repository root. Each command reads its settings from the environment and .env (see Configuration).

uv run ratchet --help
uv run ratchet <command> --help

Commands that can spend money

With RATCHET_UPSTREAM=live, these commands call Token Factory and cost money: seed-demo, bless, agent run, agent start, replay, simulate, digest run, verify-live, bench build and bench run. seed-demo prints its estimate first. verify-live, bench build and bench run ask for confirmation or --yes. Every call still goes through the spending caps. With RATCHET_UPSTREAM=sim, nothing is spent.

Common tasks

Goal	Command
Check the install	uv run ratchet doctor
Seed the demo agent and its 25 goldens	uv run ratchet seed-demo
Start everything (dashboard, proxy, mock tools, orchestrator)	uv run ratchet serve
Run a seeded scenario without the browser	uv run ratchet simulate 1 --headless
Run one task on your agent and bless it	uv run ratchet agent run "<task>" then uv run ratchet bless --last
See what is in flight, what was spent today and its prompt-cache share	uv run ratchet status
Stage timings of one verdict	uv run ratchet status --timing <verdict_id>
Undo an edit	uv run ratchet rollback <verdict_id>
Ship a held edit	uv run ratchet approve <verdict_id>
Accept that the world changed	uv run ratchet rebless <golden_run_id> <replay_run_id>
Delete old traces	uv run ratchet purge --before 2026-10-01

Exit codes

• doctor exits 1 when any check fails. Warnings don't fail it.
• simulate exits 0 only when the decision matches the scenario's expected decision, so you can use it as a smoke test (the deploy script does).
• verify-live exits 1 when any live check fails, or when you decline the cost prompt.
• bench validate exits 1 when the dataset has problems.
• Errors print error <code>: <message>, using the stable codes listed in the error code table.

Every command

Generated section

Generated by scripts/gen_cli_docs.py. Do not edit it by hand: change the source, then run uv run python scripts/gen_docs.py.

Ratchet: regression CI for self-improving agents.

Command	Does
doctor	Check settings, keys, model ids, runner, sandbox, Tavily, git, disk and caps.
serve	Dashboard (8080), proxy (8081), mock tools (8082) and the orchestrator in one process.
seed-demo	Initialize the demo skills repo and record + bless the 25 demo goldens.
verify-live	Live model and Tavily checks with real keys (about $0.05); appends to docs/VERIFIED.md.
bless	Make a live run golden (PII pass, world facts, side effects, criteria).
status	Edits in flight, the last verdicts, today's spend and its prompt-cache share.
logs	Events and worker logs for one edit.
metrics	Product metrics (or the demo funnel).
replay	Re-run an edit from selection (a fresh attempt) and wait for its verdict.
rollback	Revert the edit's commit on candidate; main is left unchanged.
approve	Merge a held edit into main.
rebless	Make the replay the new golden; archive the old one (reason: drift).
simulate	Run a seeded scenario; exit 0 iff the decision matches the expected one (every run with --repeat).
purge	Delete live and replay traces older than the date (goldens only when asked), then cassettes recorded before it that no remaining trace references.
agent	Run the agent on the control plane.
digest	Weekly digest.
db	Database.
schemas	JSON schemas.
api	Dashboard API.
bench	SkillRegress-Bench: build, label, run, compare.

ratchet doctor

Check settings, keys, model ids, runner, sandbox, Tavily, git, disk and caps.

ratchet doctor [OPTIONS]

ratchet serve

Dashboard (8080), proxy (8081), mock tools (8082) and the orchestrator in one process.

ratchet serve [OPTIONS]

Option	Type	Default	Description
--no-watch	flag	off	Do not watch the skills repo.

ratchet seed-demo

Initialize the demo skills repo and record + bless the 25 demo goldens.

ratchet seed-demo [OPTIONS]

Option	Type	Default	Description
--force	flag	off	Re-record every golden.

ratchet verify-live

Live model and Tavily checks with real keys (about $0.05); appends to docs/VERIFIED.md.

ratchet verify-live [OPTIONS]

Option	Type	Default	Description
--yes	flag	off	Skip the cost prompt.

ratchet bless

Make a live run golden (PII pass, world facts, side effects, criteria). A run an external harness left through the proxy (steps but no trace.json) is finalized first.

ratchet bless [OPTIONS] [RUN_ID]

Argument	Type	Required	Description
RUN_ID	text	no	A live run id.

Option	Type	Default	Description
--last	flag	off	Bless the most recent live or proxy-captured run.
--all-demo	flag	off	Record and bless every demo task.

ratchet status

Edits in flight, the last verdicts, today's spend and its prompt-cache share.

ratchet status [OPTIONS]

Option	Type	Default	Description
--timing	text	—	Stage timeline of a verdict.

ratchet logs

Events and worker logs for one edit.

ratchet logs [OPTIONS]

Option	Type	Default	Description
--edit	text	required	The edit id.

ratchet metrics

Product metrics (or the demo funnel).

ratchet metrics [OPTIONS]

Option	Type	Default	Description
--since	text	7d	Window, e.g. 7d or 24h.
--demo	flag	off	The anonymous demo funnel instead.

ratchet replay

Re-run an edit from selection (a fresh attempt) and wait for its verdict.

ratchet replay [OPTIONS] EDIT_ID

Argument	Type	Required	Description
EDIT_ID	text	yes	—

Option	Type	Default	Description
--runner	text	—	local or nebius for this replay.

ratchet rollback

Revert the edit's commit on candidate; main is left unchanged.

ratchet rollback [OPTIONS] VERDICT_ID

Argument	Type	Required	Description
VERDICT_ID	text	yes	—

ratchet approve

Merge a held edit into main.

ratchet approve [OPTIONS] VERDICT_ID

Argument	Type	Required	Description
VERDICT_ID	text	yes	—

ratchet rebless

Make the replay the new golden; archive the old one (reason: drift).

ratchet rebless [OPTIONS] GOLDEN REPLAY

Argument	Type	Required	Description
GOLDEN	text	yes	—
REPLAY	text	yes	—

ratchet simulate

Run a seeded scenario; exit 0 iff the decision matches the expected one (every run with --repeat). Works whatever RATCHET_SIMULATE_ENABLED says (it gates the HTTP route).

ratchet simulate [OPTIONS] SCENARIO

Argument	Type	Required	Description
SCENARIO	text	yes	—

Option	Type	Default	Description
--headless	flag	off	Print stages and the headline.
--record-fixture	flag	off	Save for replayed mode.
--repeat	int range	1	Run N times (fresh session each); report k/N.
--timeout	float range	300.0	Seconds to wait for each verdict (Jobs: raise it).

ratchet purge

Delete live and replay traces older than the date (goldens only when asked), then cassettes recorded before it that no remaining trace references.

ratchet purge [OPTIONS]

Option	Type	Default	Description
--before	text	required	YYYY-MM-DD (UTC).
--include-golden	flag	off	—

ratchet agent

Run the agent on the control plane.

Command	Does
run	Run one task live through the proxy and mock tools (record mode).
start	Run the seeded demo agent's next task.

ratchet agent run

Run one task live through the proxy and mock tools (record mode).

ratchet agent run [OPTIONS] TASK

Argument	Type	Required	Description
TASK	text	yes	The task for the agent.

ratchet agent start

Run the seeded demo agent's next task.

ratchet agent start [OPTIONS]

Option	Type	Default	Description
--demo	flag	off	Run the demo agent's next task.

ratchet digest

Weekly digest.

Command	Does
run	Write this week's digest now.

ratchet digest run

Write this week's digest now.

ratchet digest run [OPTIONS]

ratchet db

Database.

Command	Does
migrate	Apply database migrations (forward-only, idempotent).

ratchet db migrate

Apply database migrations (forward-only, idempotent).

ratchet db migrate [OPTIONS]

ratchet schemas

JSON schemas.

Command	Does
export	Write the data-contract model schemas to schemas/*.json.

ratchet schemas export

Write the data-contract model schemas to schemas/*.json.

ratchet schemas export [OPTIONS]

Option	Type	Default	Description
--out	text	schemas	—

ratchet api

Dashboard API.

Command	Does
export-openapi	Print the dashboard API's OpenAPI document (> schemas/openapi.json).

ratchet api export-openapi

Print the dashboard API's OpenAPI document (> schemas/openapi.json).

ratchet api export-openapi [OPTIONS]

ratchet bench

SkillRegress-Bench: build, label, run, compare.

Command	Does
validate	Check the dataset: skills (agentskills.io), world, workspaces, trigger routing.
build	Record + bless goldens, generate controls, inject R1–R6 (dataset construction).
run	Run one configuration over the labeled edits; write results.json and report.md.
report	Re-render report.md and cost_vs_recall.svg from results.json (no model calls).
compare	Metric deltas (b − a) per configuration present in both.
label	Show an edit's diff and labels and record a human verdict (labeling step 4).
smoke	Bench smoke test on the offline simulator: build, run every config, write the report.

ratchet bench validate

Check the dataset: skills (agentskills.io), world, workspaces, trigger routing.

ratchet bench validate [OPTIONS]

Option	Type	Default	Description
--bench	path	bench	The bench dataset directory.

ratchet bench build

Record + bless goldens, generate controls, inject R1–R6 (dataset construction).

ratchet bench build [OPTIONS]

Option	Type	Default	Description
--bench	path	bench	The bench dataset directory.
--work	path	var/bench	Bench work area (its own store and repo).
--goldens, --no-goldens	flag	--goldens	—
--controls, --no-controls	flag	--controls	—
--inject, --no-inject	flag	--inject	—
--skills	text	—	Comma-separated skills (default: all).
--types	text	—	Comma-separated R types to inject.
--force	flag	off	Redo goldens and labeled edits.
--yes, -y	flag	off	Confirm live model spend without a prompt.

ratchet bench run

Run one configuration over the labeled edits; write results.json and report.md.

ratchet bench run [OPTIONS]

Option	Type	Default	Description
--config	path	required	bench/configs/<name>.yaml
--out	path	required	Results directory, e.g. bench/results/<date>/
--bench	path	bench	The bench dataset directory.
--work	path	var/bench	Bench work area (its own store and repo).
--limit	integer	—	Run at most N edits (seeded, by type).
--types	text	—	Comma-separated types, e.g. R2,R4,N0.
--seed	integer	0	Seed for the --limit sample.
--live-tavily	flag	off	Re-query Tavily live.
--concurrency	integer	—	Edits in flight (default: config).
--yes, -y	flag	off	Confirm live model spend without a prompt.

ratchet bench report

Re-render report.md and cost_vs_recall.svg from results.json (no model calls).

ratchet bench report [OPTIONS] OUT

Argument	Type	Required	Description
OUT	path	yes	Results directory.

Option	Type	Default	Description
--bench	path	bench	The bench dataset directory.
--edits	path	—	Edits dir (default: <bench>/edits).

ratchet bench compare

Metric deltas (b − a) per configuration present in both.

ratchet bench compare [OPTIONS] A B

Argument	Type	Required	Description
A	path	yes	Baseline results directory.
B	path	yes	Results directory to compare.

Option	Type	Default	Description
--out	path	—	Also write the table to this file.

ratchet bench label

Show an edit's diff and labels and record a human verdict (labeling step 4).

ratchet bench label [OPTIONS] [EDIT_ID]

Argument	Type	Required	Description
EDIT_ID	text	no	Edit to review (default: next pending).

Option	Type	Default	Description
--bench	path	bench	The bench dataset directory.
--agree, --disagree	flag	—	Your verdict.
--decision	text	—	Corrected decision on --disagree.
--pair	text	—	task_id=regression|pass|drift (repeatable).
--reviewer	text	reviewer	Your name or handle.
--note	text	empty	Free-text note.
--status	flag	off	Only list the review queue.

ratchet bench smoke

Bench smoke test on the offline simulator: build, run every config, write the report.

ratchet bench smoke [OPTIONS]

Option	Type	Default	Description
--out	path	bench/results/smoke-sim	Smoke results dir.
--work	path	var/bench-smoke	Smoke work area.
--bench	path	bench	The bench dataset directory.
--configs	text	—	Comma-separated configs (default: all).