mirror of
https://github.com/nexu-io/open-design.git
synced 2026-05-31 19:04:39 +07:00
da19ff3ca0
* feat(mocks): replay-based mock CLIs for opencode/claude/codex/deepseek/qwen/grok
Drops in a `mocks/` top-level dir that pretends to be the real agent
CLIs by streaming pre-recorded sessions in each CLI's native stdout
protocol. Zero LLM tokens.
## Use cases
- **E2E tests** in `apps/daemon/tests/` — exercise the full chat-server
pipeline against a known trace, assert UI events / artifacts.
- **Self-validation during dev** — iterate on `claude-stream.ts` /
`json-event-stream.ts` parser changes without burning provider budget.
- **Regression harness** — replay the same trace before and after a
charter / parser change; diff the daemon events the UI surfaces.
- **Demo / onboarding** — show what a 17-tool claude editing session
looks like end-to-end, offline.
## How
- 6 bash wrappers (`mocks/bin/`) shadow the real CLIs when PATH-overlaid.
- `mocks/mock-agent.mjs` reads `mocks/recordings/<trace>.jsonl`, picks
one via env var (`SYNCLO_EXPLORE_MOCK_TRACE` / `_POOL` /
`_BY_PROMPT_HASH`), streams the trace in the requested format.
- Each format renderer matches the EXACT JSON shape the OD daemon
parser expects, verified line-by-line against
`apps/daemon/src/{json-event-stream,claude-stream}.ts`:
| CLI | streamFormat | parser source |
| ------------------------- | ------------------------- | ------------------------------------------ |
| `opencode` | `json-event-stream` | `handleOpenCodeEvent` |
| `codex` | `json-event-stream` | `handleCodexEvent` |
| `claude` | `claude-stream-json` | `createClaudeStreamHandler` |
| `deepseek` `qwen` `grok` | `plain` | `server.ts` (raw stdout) |
## Quick start
```bash
export PATH="$PWD/mocks/bin:$PATH"
export SYNCLO_EXPLORE_MOCK_TRACE=04097377 # 8-char prefix OK
export SYNCLO_EXPLORE_MOCK_NO_DELAY=1
echo "any prompt" | opencode run
echo "any prompt" | claude -p --output-format=stream-json
echo "any prompt" | codex exec
```
The mock binary announces the picked trace id on stderr:
`[mock-opencode] picked 04097377… via fixed`.
Recording selection (env, in priority order):
- `SYNCLO_EXPLORE_MOCK_TRACE=<id>` — fixed (prefix OK)
- `SYNCLO_EXPLORE_MOCK_BY_PROMPT_HASH=1` + stdin prompt — `sha256(prompt) % N`
- `SYNCLO_EXPLORE_MOCK_POOL=<tag>` — random within `agent:claude` /
`skill:agent-browser` / `outcome:failed` / etc.
- (default) uniform random
- `SYNCLO_EXPLORE_MOCK_SEED=<str>` — reproducible "random"
- `SYNCLO_EXPLORE_MOCK_NO_DELAY=1` — skip inter-event waits
## Dataset
179 anonymized Langfuse traces from this project's own production
telemetry:
- 9 agents: claude 57 · opencode 41 · codex 38 · gemini 25 ·
cursor-agent 11 · qwen 2 · copilot 2 · deepseek 2 · antigravity 1
- outcomes: succeeded 144 · failed 35
- skills: default 71 · ad-creative 50 · algorithmic-art 30 ·
agent-browser 22 · video-hyperframes 2 · plus magazine-web-ppt /
brainstorming / data-report / penpot-flutter-design-source 1 each
- 124 multi-turn (sessions with ≥2 turns)
- 18 produce `<artifact>` output
- ~4.5 MB on disk total
Anonymization: `/Users/<name>/` → `${HOME}/`,
`C:\Users\<name>\` → `%USERPROFILE%\`, project UUIDs →
stable `proj-001`, `proj-002`, …. Tool input/output payloads
preserved verbatim (templated UI, no cell-level PII).
## Smoke test
`bash mocks/scripts/smoke-test.sh` — 6 checks across all 6 agents.
All pass on this branch (verified locally):
```
✓ opencode first event = step_start
✓ codex first event = thread.started
✓ claude first event = system
✓ deepseek emitted plain text (144 chars on first line)
✓ qwen emitted plain text (144 chars on first line)
✓ grok emitted plain text (144 chars on first line)
All mock CLIs working. ✅
```
## Adding more recordings
The exporter that produced this set lives in
[nexu-io/agent-pr-explore](https://github.com/nexu-io/agent-pr-explore)
(see `cli/src/local/orchestrator/langfuse-import.ts` + the `local
langfuse-import` CLI command). Operators with the Langfuse keys can pull
more by tag / outcome / artifact / multi-turn filter, then run
`local recordings anonymize --out-dir ~/Documents/open-design/mocks/recordings`.
`mocks/README.md` has the full instructions.
## Out of scope (follow-ups)
- **ACP agents** (`devin`, `hermes`, `kilo`, `kimi`, `kiro`, `vibe`) need
a JSON-RPC server on stdio rather than a one-shot stream — separate
`format-acp.mjs` module not yet written.
- **Per-agent json-event-stream variants** (`cursor-agent`, `gemini`,
`qoder`, `copilot`, `pi`) currently fall back to the `plain` renderer;
their parsers are in `apps/daemon/src/json-event-stream.ts` and follow
the same template as `format-codex.mjs`.
## AGENTS.md updates
- Added `mocks/` to the top-level content directories listing
- Added a Validation strategy bullet pointing here for agent-stream /
parser changes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mocks): add opencode-cli/kiro-cli/vibe-acp bin aliases and unref ACP timeout
- Add mocks/bin/opencode-cli, kiro-cli, vibe-acp wrappers for the primary
RuntimeAgentDef bin names OD resolves before any fallback. Without these,
a PATH-overlaid OD daemon run bypasses the mock entirely (opencode-cli,
kiro-cli) or cannot find the mock at all (vibe-acp, which has no fallback).
- Include opencode-cli, kiro-cli, vibe-acp in the smoke-test ACP/JSON loop
so coverage is verified end-to-end.
- Call .unref() on the 30s safety timeout in format-acp.mjs so a completed
ACP session exits promptly instead of waiting the full 30 seconds.
Generated-By: looper 0.9.2 (runner=fixer, agent=claude-code)
* feat(mocks): add vela (AMR) — login / models / ACP with strict set_model gate
Extends mocks/ to cover OD's own AMR runtime. `vela` is the bin name
`apps/daemon/src/runtimes/defs/amr.ts` specifies (`bin: 'vela'`,
`streamFormat: 'acp-json-rpc'`). It's richer than the generic ACP
agents — covers full login + models + chat-session lifecycle.
### What vela does (mirrored from apps/daemon/tests/fixtures/fake-vela.mjs)
1. `vela login` — writes ~/.amr/config.json with a fake profile (controlKey,
runtimeKey, user{email,name,plan}, profile-specific apiUrl/linkUrl).
The on-disk projection is what OD's daemon login route + AmrLoginPill
poller read; production goes through device-auth, the mock skips
straight to the file write.
2. `vela models` — prints the production-shaped public model catalog as
newline-separated `public_model_* vela` lines. Override via
FAKE_VELA_MODELS env.
3. `vela agent run --runtime opencode` — ACP JSON-RPC server with three
vela-specific protocol extensions:
a. `initialize` response carries `agentCapabilities`
(`promptCapabilities.embeddedContext`) + `models`
(`currentModelId` + `availableModels`).
b. `session/new` response carries the same `models` block.
c. **Strict set_model gate**: `session/prompt` is rejected with
JSON-RPC -32602 ("session/set_model must be called before
session/prompt") UNLESS `session/set_model` (or
`session/set_config_option`) has been called for the current
sessionId. Mirrors real vela 0.0.1 contract; catches regressions
in `attachAcpSession` that silently skip set_model.
### Error injection envs (in sync with fake-vela.mjs)
FAKE_VELA_SESSION_ID - sessionId returned by session/new
FAKE_VELA_TEXT - override assistant text
FAKE_VELA_THOUGHT - optional thought_chunk before text
FAKE_VELA_SESSION_NEW_ERROR - fail session/new
FAKE_VELA_SET_MODEL_ERROR - fail session/set_model
FAKE_VELA_PROMPT_ERROR - fail session/prompt
FAKE_VELA_REQUIRE_SET_MODEL='0' - disable the strict gate (legacy)
FAKE_VELA_LOGIN_USER_EMAIL - email written into config profile
FAKE_VELA_LOGIN_USER_PLAN - plan written into config profile
FAKE_VELA_LOGIN_DELAY_MS - sleep before write (test in-flight)
FAKE_VELA_LOGIN_FAIL - print + exit 1
FAKE_VELA_MODELS - override models stdout
VELA_PROFILE - profile slot (prod | test | local)
### Components
`mocks/lib/format-vela.mjs` (~205 LOC)
- Full ACP server with vela protocol extensions
- Strict set_model gate
- Error injection plumbing
`mocks/lib/vela-subcommands.mjs` (~90 LOC)
- runVelaLogin() — writes ~/.amr/config.json
- runVelaModels() — prints catalog
`mocks/bin/vela` — dispatcher wrapper. Forwards `vela <subcmd>` to
mock-agent.mjs which routes to login/models or falls through to ACP.
`mocks/mock-agent.mjs` — parseArgs now collects positionals so the vela
dispatcher can read subcommand from there; switch case added for vela.
`mocks/scripts/smoke-test.sh` — +4 assertions:
vela models prints ≥10 catalog lines
vela login writes ~/.amr/config.json with the requested email
vela agent run ACP roundtrip (initialize+models+set_model+stream+result)
vela strict set_model gate rejects prompt without prior set_model
### Verified locally
✓ vela models printed 15 catalog lines
✓ vela login wrote ~/.amr/config.json with profile.prod.user.email
✓ vela agent run ACP roundtrip (initialize+models, set_model accepted, prompt streamed)
✓ vela strict set_model gate rejects session/prompt without prior set_model
All 21 smoke checks pass (up from 17 with previous P3 ACP commit).
### AGENTS.md + README updates
AGENTS.md — mention `vela (AMR — vela CLI)` alongside ACP agents in
the directory listing entry.
mocks/README.md — protocol table row + dedicated vela section with
subcommand contract, strict gate explanation, env-injection cheat
sheet. Mock-tree listing updated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mocks): honor REPORT_FILE env when --report-file flag not given
Harnesses that spawn the mock without translating their report-path
contract to the mock's CLI flag (notably nexu-io/agent-pr-explore's
orchestrator, which passes REPORT_FILE as env per the existing
opencode/claude/codex agent launchers) wouldn't get a report file
written, so the harness's "agent exit 0 but produced no report"
check would always fire and mark mock runs as failure even though the
stdout stream was complete.
Fix: in mock-agent.mjs parseArgs, fall through to process.env.REPORT_FILE
when --report-file wasn't provided on argv. Each format renderer already
accepts opts.reportFile and writes the recording's final assistant text
to it (`format-*.mjs` already had this — only the wiring was missing).
Verified: synclo-explore run with `mock=true, mock_trace=04097377`
against the opencode wrapper now produces a plan.md with the recording's
17-tool claude editing session report. ~1.5s per run vs ~70s real opencode.
* mocks: move recordings to Cloudflare R2; PR→main→Action upload path
The 179-recording corpus (~4.5 MB raw, ~280 KB after compression) has
been moved off git into Cloudflare R2 at the bucket open-design-mocks
under recordings/v1/. The repo now ships:
- mocks/manifest.json — the canonical catalog (renamed from
recordings/index.json) with sha256 + storage hints; consumers
fetch this to discover what exists, then pull individual jsonl
files on demand
- mocks/scripts/fetch-recordings.sh — parallel, sha256-verified,
idempotent puller for the public r2.dev URL
- mocks/scripts/add-recording.sh — local maintainer helper that
validates a new .jsonl and copies it into recordings-staging/
(no R2 calls; no credentials needed)
- mocks/scripts/upload-to-r2.mjs — called only by the CI workflow
- mocks/scripts/lib/manifest-utils.mjs — shared sha256/meta/
rebuild-histograms logic, used by both add-recording (preview)
and upload-to-r2 (actual write) so the entry shape never drifts
- .github/workflows/sync-mocks-to-r2.yml — fires on push to main
when mocks/recordings-staging/ changes; uploads to R2, updates
manifest, commits cleanup back; serialized via concurrency group
Trust model: R2 write credentials (CLOUDFLARE_API_TOKEN,
CLOUDFLARE_ACCOUNT_ID) are repo secrets; nobody can push from a
laptop. Read stays public via the r2.dev URL.
Why not pnpm install integration: contributors who do not touch
agent code do not pay the fetch cost. Fetch happens on first
smoke-test run (auto-fallback) or when a mock spawn needs data.
Repo size: -4.55 MB net (delete 179 jsonl, +280 KB manifest +
scripts). Smoke test (21 checks) still green against the fetched
corpus.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* mocks: scope R2 write token to a dedicated secret name
Use CLOUDFLARE_R2_MOCKS_TOKEN (instead of reusing the shared
CLOUDFLARE_API_TOKEN that landing-page-*.yml uses for Pages deploys)
so the R2 write capability can be scoped to just the
open-design-mocks bucket without bleeding extra capability into the
Pages workflows.
Also hardcode the powerformer CF account_id directly in the workflow
(account IDs are not secret and the shared CLOUDFLARE_ACCOUNT_ID
secret may point at a different account).
Workflow now fails fast with an actionable error message + dashboard
link if the secret is unset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* mocks: switch R2 sync to S3-compat API (wrangler getMemberships gate)
wrangler 4.x calls /memberships before any r2 action, requiring
user:read scope. R2 "Object Read & Write" tokens deliberately lack
that scope (defense in depth — a leaked token should not enumerate
account-level resources). The workflow now uses the aws CLI talking
straight to the R2 S3-compatible endpoint with SigV4, no membership
lookup.
Secret rotation: CLOUDFLARE_R2_MOCKS_TOKEN (Bearer) is replaced by
CLOUDFLARE_R2_MOCKS_AK / CLOUDFLARE_R2_MOCKS_SK (matching the
existing CLOUDFLARE_R2_RELEASES_AK/SK naming convention). End-to-end
tested locally: PUT recording → manifest rebuild → manifest PUT →
staging cleanup all green.
aws CLI is pre-installed on ubuntu-latest, so no install step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* mocks: scrub synclo namespace; use OD_MOCKS_* env prefix throughout
These mocks were copy-pasted from synclo-explore, where they
originated, and inherited the SYNCLO_EXPLORE_MOCK_* env-var
convention. That brand-bleed is not appropriate in OD: rename the
public env surface to OD_MOCKS_* (matching OD-native prefixes like
OD_MOCKS_CACHE_DIR, OD_TRACE_R2_UPLOAD, OD_EXPECT_TIMEOUT_SECONDS).
Renames:
SYNCLO_EXPLORE_MOCK_TRACE → OD_MOCKS_TRACE
SYNCLO_EXPLORE_MOCK_BY_PROMPT_HASH → OD_MOCKS_BY_PROMPT_HASH
SYNCLO_EXPLORE_MOCK_POOL → OD_MOCKS_POOL
SYNCLO_EXPLORE_MOCK_SEED → OD_MOCKS_SEED
SYNCLO_EXPLORE_MOCK_NO_DELAY → OD_MOCKS_NO_DELAY
SYNCLO_EXPLORE_MOCK_RECORDINGS_DIR → OD_MOCKS_RECORDINGS_DIR
SYNCLO_EXPLORE_MOCK_SMOKE_TRACE → OD_MOCKS_SMOKE_TRACE
SYNCLO_OD_MOCKS_I_KNOW_WHAT_IM_DOING → OD_MOCKS_ALLOW_LOCAL_UPLOAD
Also drop the inline harvester usage from README. The harvester is an
external CLI in nexu-io/agent-pr-explore — its README is the right
place for langfuse-import flags, anonymization options, etc. OD only
documents its own staging→PR→Action workflow.
Smoke test (21 checks) still green; OD_MOCKS_TRACE end-to-end
verified to route correctly.
Consumers of the OLD env names (notably the orchestrator in
nexu-io/agent-pr-explore) need a matching rename. No back-compat
shim here — the explore side has zero external users today and a
one-line follow-up is cleaner than a permanent deprecation layer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* AGENTS.md: align mock env names with mocks/ rename (SYNCLO_* → OD_MOCKS_*)
Missed in the prior commit (
|
||
|---|---|---|
| .. | ||
| format-acp.mjs | ||
| format-claude.mjs | ||
| format-codex.mjs | ||
| format-cursor-agent.mjs | ||
| format-gemini.mjs | ||
| format-opencode.mjs | ||
| format-plain.mjs | ||
| format-vela.mjs | ||
| recording-picker.mjs | ||
| vela-subcommands.mjs | ||