open-design/mocks

mocks/ — replay-based mock CLIs for OD's supported agents

A drop-in replacement for the real agent CLIs (claude, opencode, codex, gemini, cursor-agent, deepseek, qwen, grok, the ACP family devin / hermes / kilo / kimi / kiro / vibe, and the AMR vela CLI) that replays pre-recorded sessions in each CLI's native protocol — stdout streaming for most, JSON-RPC over stdio for ACP and AMR. Zero LLM tokens.

Used by:

• E2E tests in apps/daemon/tests/ — run the full chat-server pipeline against a known agent trace, assert UI events / artifacts.
• Local self-tests during development — iterate on chat-routes.ts, claude-stream.ts, json-event-stream.ts parser changes without burning provider budget.
• Demo / onboarding — show what a 17-tool claude editing session looks like end-to-end, offline.
• Regression harness — replay the same trace before and after a charter / parser change; diff the events the daemon surfaces.

The recordings are anonymized exports from open-design's Langfuse project (179 traces across 9 agents and 5+ skills as of this commit).

---

tl;dr

# First-time setup — pull the recording corpus from R2 (~30s, 4.5MB):
bash mocks/scripts/fetch-recordings.sh
# Subsequent runs hit the local cache (sha256-verified, instant).

# Make the mock CLIs override the real ones for this shell:
export PATH="$PWD/mocks/bin:$PATH"

# Pick any recording to play back (8-char prefix OK):
export OD_MOCKS_TRACE=04097377

# Speed up replay (skip inter-event sleeps):
export OD_MOCKS_NO_DELAY=1

# Now anything that spawns opencode/claude/codex gets the recording:
echo "any prompt body" | opencode run
echo "any prompt"     | claude -p --output-format=stream-json
echo "any prompt"     | codex exec

The mock binaries are bash wrappers that exec node mocks/mock-agent.mjs --as <agent>. Anything fed to stdin is discarded by the renderer but used by the recording picker (see hash mode below).

Recordings live on R2, not in this repo

The 179-recording corpus (~4.5 MB) is hosted on Cloudflare R2 at open-design-mocks and fetched on demand — pnpm install does NOT pull them, and the repo stays small. Recordings only land in mocks/recordings/ when:

1. You run bash mocks/scripts/fetch-recordings.sh directly, OR
2. bash mocks/scripts/smoke-test.sh runs and the dir is empty (auto- fetch fallback), OR
3. A mock binary spawn finds no data — it errors with a pointer at the fetch script (no silent failure).

This is by design: contributors who don't touch agent code don't pay the fetch cost. CI jobs that DO touch agent code (apps/daemon/tests/ parser changes, etc.) run the fetch as a quick pre-step and cache mocks/recordings/ between runs.

# Fetch everything (parallel, sha256-verified, idempotent):
bash mocks/scripts/fetch-recordings.sh

# Fetch a subset:
bash mocks/scripts/fetch-recordings.sh --agent claude       # 57 claude traces
bash mocks/scripts/fetch-recordings.sh --outcome failed     # 35 failed-path traces
bash mocks/scripts/fetch-recordings.sh --skill agent-browser

# Override cache location (e.g. share across multiple OD checkouts):
OD_MOCKS_CACHE_DIR=~/.cache/od-mocks bash mocks/scripts/fetch-recordings.sh

Manifest at mocks/manifest.json is the committed source of truth — it lists every recording's trace_id, sha256, bytes, agent, outcome, skills, multi_turn, plus histograms over the corpus. Tooling reads this; you don't have to.

Provenance per recording

Beyond identity (trace_id, sha256), each manifest entry carries fixture-trust signals so consumers can decide whether the recording is still meaningful as the real CLIs evolve:

Field	Meaning
captured_at	ISO 8601 timestamp of the original session — populated for all 179 current entries
cli_version	The CLI version the trace was captured against (e.g. "claude-code 1.0.65") — populated only on traces the harvester writes it to, null otherwise
protocol_version	Stream-format version ("claude-stream-json/v1", "opencode/json-event-stream") — populated by harvester
anonymization_version	Which anonymizer pass scrubbed the recording — populated by harvester

For now most of these are null on the existing 179 — the harvester in nexu-io/agent-pr-explore is the next thing to teach to write them. Once a recording's cli_version falls behind the actual CLI by more than one minor version, treat it as a candidate for re-harvest.

Golden daemon-event snapshots

mocks/golden/<trace>.events.json holds the exact event sequence the OD daemon emits when fed each (mock CLI → handler) pipeline. Diffed on every pnpm --filter @open-design/daemon test run by apps/daemon/tests/mocks-golden.test.ts.

A parser refactor that semantically changes events (drops a field, renames sessionId, stops emitting turn_end) fails the diff loudly. After an intentional parser change, regenerate:

MOCKS_GOLDEN_UPDATE=1 pnpm --filter @open-design/daemon test mocks-golden
git diff mocks/golden/    # eyeball the new shapes
git add mocks/golden/ && git commit -m "mocks: refresh goldens for <parser change>"

Per-spawn volatile fields (currently just claude's generated sessionId) are stripped to "<normalized>" so the snapshot stays stable. See mocks/golden/README.md for the coverage rationale.

Real-CLI contract check

The mocks catch parser regressions against the recordings; they do not catch the recordings themselves drifting away from the live agent CLIs. For that, mocks/scripts/contract-check.sh spawns a real CLI alongside the mock with a fixed prompt and prints a side-by-side event-type distribution.

This is human-driven and costs real LLM tokens — run on a real-CLI release or before a parser refactor, not on a cron. Full doc: docs/MOCKS-CONTRACT-CHECK.md.

---

What gets emitted

Each renderer matches the EXACT event shapes the OD daemon expects, as verified line-by-line against the parsers in apps/daemon/src/:

CLI	OD streamFormat	Parser source
opencode	json-event-stream (opencode kind)	json-event-stream.ts:handleOpenCodeEvent
codex	json-event-stream (codex kind)	json-event-stream.ts:handleCodexEvent
claude	claude-stream-json	claude-stream.ts:createClaudeStreamHandler
gemini	json-event-stream (gemini kind)	json-event-stream.ts:handleGeminiEvent
cursor-agent	json-event-stream (cursor-agent kind)	json-event-stream.ts:handleCursorEvent
deepseek qwen grok	plain	server.ts (raw stdout = final assistant text)
devin hermes kilo kimi kiro vibe	acp-json-rpc	acp.ts:attachAcpSession
vela (AMR)	acp-json-rpc + login / models subcommands	runtimes/defs/amr.ts + apps/daemon/tests/fixtures/fake-vela.mjs (sibling stub)

Note on gemini and cursor-agent: OD's parsers for these two agents do NOT recognize tool-call events — only init / assistant text / usage. The renderers therefore emit ONLY the final assistant text wrapped in the expected init/text/usage envelope. Tool calls present in the source recording are silently dropped (which matches the real CLI's UI behavior — these agents don't surface tools in OD's chat view).

Note on ACP agents (devin / hermes / kilo / kimi / kiro / vibe): These do NOT stream stdout — they speak JSON-RPC v2 over stdio. OD's daemon sends initialize → session/new → (optional session/set_model) → session/prompt; the mock responds in order, streams text via session/update notifications carrying agent_message_chunk parts, then responds to the prompt request with usage stats. Tool calls aren't part of the ACP protocol on this path (tools surface via MCP or other side channels), so they're dropped from playback.

Note on vela (AMR): vela is the bin OD's AMR runtime spawns. It extends the generic ACP shape with agentCapabilities + models blocks in initialize / session/new, plus a strict set_model gate — session/prompt is rejected with -32602 until session/set_model (or session/set_config_option) has been called for the current sessionId, mirroring real vela 0.0.1 contract.

vela also has two non-ACP subcommands:

• vela login → writes ~/.amr/config.json with a fake profile so OD's daemon login route + AmrLoginPill poller see the same on-disk projection production produces.
• vela models → prints the production-shaped public_model_* vela catalog.

Error injection envs (kept in sync with apps/daemon/tests/fixtures/fake-vela.mjs): FAKE_VELA_SESSION_NEW_ERROR / FAKE_VELA_SET_MODEL_ERROR / FAKE_VELA_PROMPT_ERROR / FAKE_VELA_LOGIN_FAIL / FAKE_VELA_REQUIRE_SET_MODEL=0.

Each tool call from the recording is rendered with the original input arguments and tool output. The agents' assistant text is rendered as the final message.

---

Recording selection

Driven by env vars, in priority order:

Env	Behavior
OD_MOCKS_TRACE=<id>	Always play this trace. 8-char prefix OK.
OD_MOCKS_BY_PROMPT_HASH=1 + stdin prompt	Deterministic by sha256(prompt) % len(all). Same prompt → same trace. Useful for "stable answer per question" tests.
OD_MOCKS_POOL=<tag>	Random within the tag pool. Examples: agent:claude, skill:agent-browser, outcome:failed.
OD_MOCKS_SEED=<str>	Makes "random" picks reproducible across runs.
OD_MOCKS_NO_DELAY=1	Skip inter-event waits.
OD_MOCKS_RECORDINGS_DIR=<path>	Override the recordings dir.

If none are set, a uniformly random recording is played each invocation.

The mock binary announces the picked trace id on stderr:

[mock-opencode] picked 04097377… via fixed

This line is invisible to OD's stdout parser but useful for "wait, why did my test get the FAQ-fix trace?" debugging.

---

Recording catalog

The recordings live as one JSONL file per Langfuse trace under recordings/. Each file starts with a meta event carrying:

{
  "type": "meta",
  "source": {"provider": "langfuse", "trace_id": "...", "project_id": "..."},
  "agent": "claude" | "codex" | "opencode" | "gemini" | "cursor-agent" | "qwen" | "copilot" | "deepseek" | "antigravity",
  "model": "...",
  "outcome": "succeeded" | "failed" | "errored" | "interrupted",
  "duration_ms": 33620,
  "tool_call_count": 17,
  "error_count": 0,
  "total_tokens": 12345,
  "tags": ["agent:claude", "skill:agent-browser", "open-design", ...],
  "user_input": "...",
  "session_id": "..."
}

Subsequent events are tool_call, tool_result, and report (the final assistant text).

Indexed metadata

mocks/manifest.json is a flat manifest with one entry per recording plus histograms over all recordings, committed to the repo. It's also mirrored to R2 alongside the .jsonl files so consumers can fetch the current catalog without cloning. Query with jq:

# All multi-turn claude sessions about HTML editing
jq '.entries[] | select(.agent=="claude" and .multi_turn==true)' \
  mocks/manifest.json | head -50

# Failed codex traces (negative-path tests)
jq '.entries[] | select(.agent=="codex" and .outcome=="failed") | .trace_id' \
  mocks/manifest.json

# Agent-browser skill, sorted by tool count desc
jq '[.entries[] | select(.skills | index("agent-browser"))] | sort_by(-.tool_count)' \
  mocks/manifest.json

Headline stats (current dataset)

Dimension	Distribution
Agents	claude 57 · opencode 41 · codex 38 · gemini 25 · cursor-agent 11 · qwen/copilot/deepseek 2 each · antigravity 1
Outcomes	succeeded 144 · failed 35
Skills	default 71 · ad-creative 50 · algorithmic-art 30 · agent-browser 22 · video-hyperframes 2 · magazine-web-ppt / brainstorming / data-report / penpot-flutter 1 each
Multi-turn	124 traces tied to a session with ≥2 turns
Artifact	18 traces produce <artifact> output

---

Anonymization

User-specific data has been scrubbed from every recording:

• /Users/<name>/…, /home/<name>/…, C:\Users\<name>\… → ${HOME}/… / %USERPROFILE%\…
• Project UUIDs → stable proj-001, proj-002, … per recording
• meta tag project:<uuid> rewritten too

The anonymizer is idempotent. Tool input/output payloads (HTML, code, etc.) are preserved verbatim — they're templated UI without cell-level PII; if a future audit finds otherwise, add specific scrubs in the harvester repo (see "Adding more recordings" below) and re-run.

---

Adding more recordings

Local maintainer flow — the .jsonl never enters the repo. Only the manifest delta (≈200 B per entry) gets committed.

Step 1 — produce an anonymized .jsonl

The harvester that produced the current 179-trace set lives in a separate repo, nexu-io/agent-pr-explore. See its README for how to authenticate against your trace store, filter by skill / agent / outcome, and anonymize the result. Output is one <trace-id>.jsonl file per recording.

Step 2 — one-shot upload + manifest update

# prereq, once: wrangler login (OAuth, no token to manage)
bash mocks/scripts/upload-recording.sh /path/to/<trace-id>.jsonl

The script validates the file, prints the manifest entry it will add, uploads the .jsonl to R2, rewrites mocks/manifest.json locally, then uploads the updated manifest to R2 too (so consumers see the new entry without waiting for the next git push).

Step 3 — commit the manifest delta

git add mocks/manifest.json
git commit -m "mocks: add recording <trace-id>"
git push     # or open a PR — your call

The only thing in the commit is a ~200-byte JSON edit listing the new entry's trace_id, sha256, bytes, agent, outcome, skills, etc. The .jsonl itself stays in R2.

Trust model

• R2 write is wrangler-OAuth gated. Maintainers do wrangler login once. The bucket is on the powerformer Cloudflare account (pinned in the script). No long-lived tokens in repo secrets, no Action to hijack — just account access.
• Repo stays small forever. No .jsonl files ever land in git; the manifest grows by ~200 B per recording.
• Read stays public. Anyone can fetch via the r2.dev URL — see Recordings live on R2, not in this repo.

Removing a recording

# 1. delete from R2
export CLOUDFLARE_ACCOUNT_ID=64ad4569ffd912432d6b86d5656484c4
wrangler r2 object delete open-design-mocks/recordings/v1/<trace-id>.jsonl --remote
# 2. drop the entry from manifest.json (edit by hand, or use `jq`)
# 3. re-upload manifest
wrangler r2 object put open-design-mocks/recordings/v1/manifest.json \
  --file mocks/manifest.json --remote
# 4. git add mocks/manifest.json && git commit && git push

There's no automation for delete because (a) it's rare and (b) you want a human to think about whether removing a recording would invalidate any test fixtures that pin it via OD_MOCKS_TRACE=<id>.

---

Usage from OD's test code

From a test (Vitest / Jest)

import { spawn } from 'node:child_process';
import { join } from 'node:path';

const MOCK_BIN = join(__dirname, '../../mocks/bin');

it('parses an opencode session with 4 tool calls into 4 UI events', async () => {
  const child = spawn('opencode', ['run'], {
    env: {
      ...process.env,
      PATH: `${MOCK_BIN}:${process.env.PATH}`,
      OD_MOCKS_TRACE: '06a9324a',   // 4-tool claude session
      OD_MOCKS_NO_DELAY: '1',
    },
    stdio: ['pipe', 'pipe', 'pipe'],
  });
  child.stdin.write('test prompt');
  child.stdin.end();
  // ... assert events parsed from child.stdout
});

From a manual playback

# See what claude's 17-tool "delete v2" session emits to OD:
export PATH=$(git rev-parse --show-toplevel)/mocks/bin:$PATH
export OD_MOCKS_TRACE=04097377
export OD_MOCKS_NO_DELAY=1
echo "anything" | claude -p --output-format=stream-json | jq .type | uniq -c

---

Files

mocks/
├── README.md                 ← you are here
├── mock-agent.mjs                ← entry; routes --as <agent> to format renderer
├── lib/
│   ├── recording-picker.mjs      ← env-driven trace selection
│   ├── format-opencode.mjs       ← matches handleOpenCodeEvent
│   ├── format-codex.mjs          ← matches handleCodexEvent
│   ├── format-claude.mjs         ← matches createClaudeStreamHandler
│   ├── format-gemini.mjs         ← matches handleGeminiEvent
│   ├── format-cursor-agent.mjs   ← matches handleCursorEvent
│   ├── format-acp.mjs            ← JSON-RPC server matching attachAcpSession
│   ├── format-vela.mjs           ← AMR vela: ACP + models block + set_model gate
│   ├── vela-subcommands.mjs      ← `vela login` + `vela models` handlers
│   └── format-plain.mjs          ← raw stdout (deepseek/qwen/grok)
├── bin/
│   ├── opencode  claude  codex
│   ├── gemini    cursor-agent
│   ├── deepseek  qwen    grok
│   ├── devin hermes kilo kimi kiro vibe
│   └── vela                       ← 15 bash wrappers, PATH-overlay
├── manifest.json                 ← committed: 179 entries' metadata + sha256 + provenance + R2 storage hints
├── golden/                       ← committed: daemon-event regression snapshots
│   ├── README.md
│   └── *.events.json             ← 3 representative traces (claude/codex/opencode)
├── scripts/
│   ├── smoke-test.sh             ← 21 checks; auto-fetches recordings if empty
│   ├── fetch-recordings.sh       ← pull from R2 (parallel, sha256-verified, idempotent)
│   ├── upload-recording.sh       ← maintainer-local: validate + wrangler put + manifest update
│   ├── contract-check.sh         ← real-CLI vs mock protocol drift check (manual)
│   └── lib/
│       └── manifest-utils.mjs    ← shared sha256 / meta-parse / manifest-rebuild logic
└── recordings/                   ← populated at runtime, gitignored .jsonl
    └── .gitignore                ← recordings come via fetch

No external dependencies. Pure node:fs/crypto/child_process. Works under any Node ≥18.

---

Limitations

• copilot, qoder, pi (the niche copilot-stream-json / qoder-stream-json / pi-rpc formats) are recorded but not yet rendered as their native protocols — they fall back to the plain renderer for now. If you need them, add a format-<agent>.mjs following the same pattern as format-codex.mjs; the parsers are in apps/daemon/src/{copilot-stream,qoder-stream}.ts and the pi-rpc handler inside apps/daemon/src/server.ts.
• The mock does not honor CLI flags that change semantics (--model, --permission-mode, --allowed-tools). They're silently ignored.

---

Provenance / safety

All recordings come from open-design's own Langfuse project (the open-design project under the powerformer org). Users opted into telemetry when they installed the desktop client. The anonymizer removed user-identifying paths and project UUIDs before checking in.

If you find a recording that includes content that should be redacted, follow the Removing a recording flow above.