open-design/apps/web/tests/analytics-configure-globals.test.ts
lefarcen 204599a7ae
feat(analytics): ship PostHog v2 event schema (#2285)
* feat(analytics): ship PostHog v2 event schema

Aligns the PostHog wire format with the product team's v2 tracking
spec (Open Design 埋点文档 2.0). The previous v1 catalogue defined a
flat per-page event name (home_view / studio_click / settings_view…);
v2 collapses everything to four core events identified through the
page_name + area + element triplet so dashboards can group by surface
without owning a separate event per page.

Key changes
- packages/contracts/src/analytics: collapse to page_view / ui_click /
  surface_view / *_result event names; bump EVENT_SCHEMA_VERSION to 2;
  rename the wire field anonymous_id → device_id (value unchanged);
  promote the configure-state triplet (has_available_configure_cli /
  configure_type / configure_availability) to a global PostHog register
  so every event inherits it without per-helper boilerplate.
- apps/web/src/analytics: rewrite the 43 trackXxx helpers behind the
  new typed catalogue; opt out of PostHog's built-in UA bot filter so
  legitimate embedded webviews, fingerprinted browsers, and the
  Playwright-based e2e runs ingest captures (the Privacy → "Share
  usage data" toggle remains the single consent gate).
- apps/web components: wire P0/P1/P2 click + view + result surfaces
  end-to-end — left nav, toolbar, home chat composer, recent projects,
  new project modal, plugins / design systems / integrations /
  automations pages, file manager, artifact toolbar/header/share popup,
  feedback panel, settings sidebar / language / appearance /
  notifications / pets / privacy / connectors. Fixes the v1 feedback
  bug where action=clear_feedback_rating shipped rating=null instead of
  the rating being cleared.
- apps/daemon: extend run_created / run_finished with the v2 context
  (entry_from / project_kind / target_platforms / fidelity /
  connectors / etc.), add explicit error_code classification on
  result=failed (run.errorCode → AGENT_SIGNAL_* → AGENT_EXIT_* →
  AGENT_TERMINATED_UNKNOWN), and read device_id from the new
  x-od-analytics-device-id header. Also moves the run_created /
  run_finished emission to the canonical /api/runs handler in
  server.ts; the chat-routes copy was shadowed by Express's earlier
  registration and never executed, which also meant run.clientType
  never made it to Langfuse — fixed in the same move.

Verification
- pnpm guard / pnpm typecheck clean for daemon, web, and contracts.
- pnpm --filter @open-design/web test: 1645/1645 passing.
- End-to-end smoke through Playwright + local PostHog ingest project
  420348: every page_view (home/projects/automations/design_systems/
  plugins/integrations/chat_panel/file_manager), every nav element,
  the new_project_modal surface_view + tab + create flow, the
  plugin_replacement_modal surface_view, settings_view across nine
  sections, settings_cli_test_result (codex CLI), the
  project_create_result success path, and run_created + run_finished
  (result=failed, error_code=AGENT_EXIT_1) all reached PostHog with
  the v2 schema and the expected device_id / page_name / area /
  element / fidelity / target_platforms props. The remaining
  *_result events (artifact_export / feedback_submit / file_upload /
  plugin_replacement / settings_byok_test / settings_connector_auth)
  are wired in code; production traffic will trigger them.

* fix(analytics): preserve style category on design-systems surface chip switch

The merge resolution in DesignSystemsTab incorrectly re-introduced a
`setCategory('All')` call alongside the new `trackDesignSystemsTopClick`
emit. main intentionally keeps the active style category when the surface
filter refines within it; the regression was caught by the existing
"keeps the style category when a surface chip refines within it" test
in tests/components/DesignSystemsTab.test.tsx.

* fix(analytics): address review — senseaudio passthrough + daemon-side configure-state

Two follow-ups from the v2 schema review on #2285:

1. `byokProtocolToTracking()` was still falling through to `null` for
   `senseaudio` even though the v2 BYOK provider enum now lists it. Every
   `SettingsDialog` BYOK call site guards on `if (byokProviderId)`, so a
   user on SenseAudio was silently dropping the provider-option,
   field-focus, and test-result captures. Added the missing case so
   SenseAudio gets the same analytics coverage as the other providers.

2. The daemon-authoritative `run_created` / `run_finished` events were
   missing the configure-state triplet (`has_available_configure_cli` /
   `configure_type` / `configure_availability`) that v2 promotes to a
   global register on the web side. Daemon captures don't go through the
   PostHog global register, so dashboards couldn't segment run lifecycle
   by execution setup after the migration.

   The fix derives the triplet server-side from `detectAgents()` and the
   request's `agentId` before `design.analytics.capture(...)`:
     - has_available_configure_cli: any CLI on PATH reports installed
     - configure_type: 'local_cli' when the run targets an installed CLI,
       otherwise 'unknown' (daemon can't see BYOK keys, which live in
       web-client storage)
     - configure_availability: 'available' / 'unavailable' / 'unknown'
       based on the requested agent's install status, with a fallback to
       'available' when any CLI is installed

   This keeps the v2 schema consistent across both daemon-side and
   web-side captures.

* fix(analytics): wire setConfigureGlobals so browser events carry fresh state

Third follow-up from the v2 schema review on #2285. The previous fix
addressed senseaudio + daemon-side configure-state, but reviewer flagged
that `setConfigureGlobals` was still defined-only — no caller — so every
browser-side capture inherited the boot defaults
(`has_available_configure_cli=false`, `configure_type='unknown'`,
`configure_availability='unknown'`). PostHog dashboards therefore could
not segment the new `page_view` / `ui_click` / `surface_view` events by
execution setup after a user configured their environment.

Changes:

- `packages/contracts/src/analytics/events.ts` — add a pure
  `deriveConfigureGlobals(mode, agentId, agents, byokConfigured)` helper
  so the web client and the daemon can derive the triplet from the same
  source of truth. The helper covers all 5 `configure_type` buckets
  (`local_cli` / `byok` / `both` / `none` / `unknown`) and the 3
  `configure_availability` buckets (`available` / `unavailable` /
  `unknown`).
- `apps/web/src/App.tsx` — add a useEffect that re-derives the triplet
  whenever the user changes execution mode, selects a new CLI, saves a
  BYOK key, or the detected-agent list refreshes, then pushes it to
  PostHog via `analytics.setConfigureGlobals(...)`. The setter goes
  through the provider so the analytics module stays the single source
  of truth.
- `apps/web/src/analytics/provider.tsx` — expose
  `setConfigureGlobals` on the analytics context and the test stub so
  consumers route through the provider boundary.
- `apps/daemon/src/server.ts` — switch the daemon-side derive in
  `/api/runs` to the shared `deriveConfigureGlobals` helper so the
  authoritative run_created/run_finished captures match the web-side
  payload. BYOK credentials live in the web client and stay invisible
  to the daemon, so the daemon arm passes `byokConfigured: undefined`
  and falls back to the installed-CLI signal.
- `apps/web/tests/analytics-configure-globals.test.ts` — new regression
  test that pins the derive behavior across all branches and confirms
  the setter actually mutates the client-side store. Locks the wire-up
  so a future refactor can't silently turn the setter back into a
  no-op.

Verification: pnpm guard clean; daemon / web typecheck clean; web tests
1703/1703 passing (up from 1696 — 7 new tests in the configure-globals
suite).

* fix(analytics): emit projects page_view + drop misattributed chat_panel source

Fourth review pass on PR #2285. Two follow-ups from mrcfps:

1. DesignsTab (projects landing) was emitting click events but no
   matching page_view. Opening /projects without clicking anything left
   the surface invisible in PostHog. Added a once-per-mount
   trackPageView({ page_name: 'projects' }) with the same ref-keyed
   pattern HomeView / PluginsView use.

2. ChatComposer was hard-coding source: 'recent_project' on every
   chat_panel page_view. The web router currently only carries
   projectId / conversationId / fileName, so we cannot distinguish a
   New-project launch from a template-pick or a Recent-projects click
   from this layer. A false constant would over-attribute every chat
   launch to 'recent_project' and break the funnel slice this schema
   was meant to unlock. Dropped the field for now — better no source
   than the wrong source — until the router grows a launch-source
   channel; the field is still defined as optional on PageViewProps so
   the channel can land in a follow-up PR.

Verification: web typecheck clean; web tests 1703/1703 passing.

* fix(analytics): correct plugin-replacement async result + heterogeneous upload + missing requestId

Three follow-ups from the fifth review pass on PR #2285:

1. **plugin_replacement_result emitted before the apply settled**
   (`apps/web/src/components/HomeView.tsx`). The modal's confirm action
   was a synchronous wrapper around an async `usePlugin(...)` call, so
   the surrounding try/catch never observed real failures and every
   attempt was reported as `result=success`. Changed `PendingReplacement.
   confirm` to return `Promise<void>`, made the wrapper return the
   underlying promise, and moved the analytics emit into an async
   IIFE in the click handler so the success/failure branches reflect
   the actual outcome.

2. **file_upload_result mis-typed heterogeneous batches**
   (`apps/web/src/components/FileWorkspace.tsx`). The earlier
   implementation only inspected `picked[0]`, so a mixed batch like
   `image.png + demo.mp4` reported `file_type=image`. Per the comment
   above the block ("mixed batches collapse to other"), the
   implementation now maps every file to a tracking type, collapses to
   `other` when more than one distinct type is present, and falls
   back to the single type otherwise.

3. **project_create_result lost the click→result correlation id**
   (`apps/web/src/components/NewProjectPanel.tsx`). The click event
   no longer carried the locally-generated `requestId` that
   `project_create_result` keeps, so the two could not be joined.
   `trackNewProjectModalElementClick()` now accepts an optional
   `{ requestId }`, mirroring the other helpers, and the create-button
   click threads the same id used for the result.

Verification: web typecheck clean; web tests 1703/1703 passing.

* fix(analytics): gate configure-state on agents probe + drop unsent run_created fields

Two follow-ups from the sixth review pass on PR #2285:

1. **Cold-start configure-state was stamped before fetchAgents() landed**
   (`apps/web/src/App.tsx`). The useEffect that pushes the v2 triplet
   into the PostHog global register fired on first paint with
   `agents=[]`, so the first home/projects/plugins page_view reported
   `has_available_configure_cli=false` / `configure_availability=
   unavailable` even on machines that did have an installed CLI. The
   effect now waits on `agentsLoading === false` and leaves the boot
   defaults ('unknown'/'unknown') in place until the probe resolves.

2. **Daemon read run-context fields the web never sends**
   (`apps/daemon/src/server.ts`). The daemon-side run_created /
   run_finished baseProps read `projectKind`, `entryFrom`,
   `projectSource`, `targetPlatforms`, `companionSurfaces`, `fidelity`,
   `connectors`, `useSpeakerNotes`, `includeAnimations`,
   `referenceTemplate`, and `aspect` from `req.body`, but
   `packages/contracts/src/api/chat.ts` and
   `apps/web/src/providers/daemon.ts` don't carry those keys on the
   wire. Reading them therefore always produced null/undefined.
   Dropped the unsent fields from the daemon capture; a follow-up can
   extend the create payload to thread the real context through. The
   `design_system_id` field stays because the chat contract does send
   it.

Tests: added 3 regression tests in `tests/analytics-configure-globals.
test.ts` covering the boot-time gating contract (empty agents +
daemon mode → unavailable / local_cli; installed agent → available;
undefined agents list → unavailable). Verification: web typecheck
clean; daemon typecheck clean; web tests 1706/1706 passing (up from
1703 — 3 new cold-start tests).

* fix(analytics): pin mode='daemon' so missing-agent run reports unavailable

Eleventh review pass on PR #2285. mrcfps flagged that
`apps/daemon/src/server.ts` was calling `deriveConfigureGlobals(...)`
without `mode`, so the helper fell through to the generic branch.
Result: a run for an uninstalled agent was tagged
`configure_availability: 'available'` whenever any OTHER CLI was on
PATH, because the generic branch only looks at the cohort-wide
"any installed?" signal. That precisely undermines the slice the
daemon emit is trying to power.

The daemon's /api/runs handler is always a daemon-mode capture
(daemon is the local CLI runner — BYOK lives in the web layer), so we
now pin `mode: 'daemon'` on the call site. The helper then judges
`configure_availability` from the REQUESTED agent's install status and
reports `unavailable` when the user picked an agent that is not
installed, even if peers are.

Added a regression case in `tests/analytics-configure-globals.test.ts`:
`{ mode: 'daemon', agentId: 'codex', agents: [{claude,true},{codex,false}] }`
→ `{ has_available_configure_cli: true, configure_type: 'local_cli',
configure_availability: 'unavailable' }`.

Verification: daemon typecheck clean; web tests 1707/1707 passing
(up from 1706 — 1 new regression test).

* fix(analytics): hoist chat_panel page_view + thread requestId

- Move chat_panel page_view emit from ChatComposer to ProjectView so
  it survives activeConversationId-driven ChatPane remounts. ProjectView
  keys the dedupe ref by project.id; the composer drops its duplicate.
- Thread { requestId } into trackAssistantFeedbackReasonSubmitClick so
  the click pairs with the existing feedback_submit_result on the same
  request id (mirrors the trackNewProjectModalElementClick pattern).

* fix(analytics): keep v2 super-props alive across reset and stamp design_system_source

- Snapshot the register payload in client.ts on PostHog init and
  re-register it from applyConsent(true) and applyIdentity() so a
  privacy-toggle or Delete-my-data rotation does not resume capture
  without event_schema_version / device_id / session_id / locale /
  configure-state globals. setConfigureGlobals() also patches the
  cache so a later restore picks up the current configure state.
- Stamp design_system_source on daemon-side run_created / run_finished
  (it is required by RunCreatedProps / RunFinishedProps). Daemon
  can't tell default vs user_selected vs inherited from the wire, so
  it derives 'unknown' when designSystemId is present, 'not_applicable'
  otherwise — a follow-up that threads designSystemSource through
  CreateRunRequest can replace this with the precise source.
2026-05-20 13:04:20 +08:00

197 lines
6.5 KiB
TypeScript

// Regression test for the v2 configure-state globals
// (has_available_configure_cli / configure_type / configure_availability).
//
// Reviewer comment on PR #2285 (mrcfps, 2026-05-19) flagged that
// `setConfigureGlobals` was defined but never called, so every browser
// capture inherited the boot defaults `{ false, 'unknown', 'unknown' }`.
// App.tsx now drives the setter from a useEffect that watches mode /
// agentId / apiKey / apiProtocolConfigs / agents; these tests pin the
// derive-then-register behavior end-to-end against the client module so a
// future refactor can't silently regress it back to a no-op setter.
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import {
deriveConfigureGlobals,
type DeriveConfigureGlobalsInput,
} from '@open-design/contracts/analytics';
import {
getConfigureGlobals,
setConfigureGlobals,
} from '../src/analytics/client';
const BOOT_DEFAULTS = {
has_available_configure_cli: false,
configure_type: 'unknown' as const,
configure_availability: 'unknown' as const,
};
describe('deriveConfigureGlobals', () => {
it('returns "none" / "unknown" when nothing is configured', () => {
expect(deriveConfigureGlobals({})).toEqual({
has_available_configure_cli: false,
configure_type: 'none',
configure_availability: 'unknown',
});
});
it('reports local_cli when an installed CLI is the selected agent in daemon mode', () => {
const input: DeriveConfigureGlobalsInput = {
mode: 'daemon',
agentId: 'claude',
agents: [
{ id: 'claude', available: true },
{ id: 'codex', available: false },
],
};
expect(deriveConfigureGlobals(input)).toEqual({
has_available_configure_cli: true,
configure_type: 'local_cli',
configure_availability: 'available',
});
});
it('marks the configure as unavailable when the selected daemon-mode agent is not installed', () => {
const input: DeriveConfigureGlobalsInput = {
mode: 'daemon',
agentId: 'codex',
agents: [
{ id: 'claude', available: true },
{ id: 'codex', available: false },
],
};
expect(deriveConfigureGlobals(input)).toMatchObject({
configure_type: 'local_cli',
configure_availability: 'unavailable',
});
});
it('reports byok when an api-mode user has saved credentials', () => {
expect(
deriveConfigureGlobals({
mode: 'api',
byokConfigured: true,
agents: [],
}),
).toEqual({
has_available_configure_cli: false,
configure_type: 'byok',
configure_availability: 'available',
});
});
it('reports both when api-mode user also has CLIs installed', () => {
expect(
deriveConfigureGlobals({
mode: 'api',
byokConfigured: true,
agents: [{ id: 'claude', available: true }],
}),
).toMatchObject({
has_available_configure_cli: true,
configure_type: 'both',
});
});
});
describe('deriveConfigureGlobals — cold-start gating', () => {
// Reviewer #2285 (mrcfps, 2026-05-20 03:10) flagged a cold-start hole:
// `fetchAgents()` is async, so the App-level useEffect first fires with
// `agents=[]` and `mode='daemon'`. Without gating, the helper used to
// stamp `has_available_configure_cli=false / configure_availability=
// unavailable` on every page_view fired before the probe resolved.
// App.tsx now waits on `agentsLoading === false` and leaves the boot
// defaults in place; these tests pin what the helper SHOULD return
// for the empty-agents / partial-config inputs so a future caller
// can't silently skip the gate again.
it('reports unavailable / local_cli when daemon mode is pinned but no agents are loaded yet', () => {
expect(
deriveConfigureGlobals({ mode: 'daemon', agentId: 'claude', agents: [] }),
).toEqual({
has_available_configure_cli: false,
configure_type: 'local_cli',
configure_availability: 'unavailable',
});
});
it('reports has_available_configure_cli=true once the probe lands and the selected agent is installed', () => {
expect(
deriveConfigureGlobals({
mode: 'daemon',
agentId: 'claude',
agents: [{ id: 'claude', available: true }],
}),
).toEqual({
has_available_configure_cli: true,
configure_type: 'local_cli',
configure_availability: 'available',
});
});
it('survives an undefined agents list (caller forgot to pass)', () => {
expect(
deriveConfigureGlobals({ mode: 'daemon', agentId: 'claude' }),
).toMatchObject({
has_available_configure_cli: false,
configure_availability: 'unavailable',
});
});
// Reviewer #2285 (mrcfps, 2026-05-20 03:36) flagged the daemon call site
// for not pinning `mode: 'daemon'`. Without the explicit mode the helper
// falls through to the generic branch and reports `available` whenever
// any unrelated CLI is on PATH — even when the requested agent is the
// one that's missing. This test pins the right behavior: a run targeted
// at an uninstalled agent must report `unavailable`, regardless of
// whether other agents happen to be installed.
it('reports unavailable when the requested daemon-mode agent is missing even if peers are installed', () => {
expect(
deriveConfigureGlobals({
mode: 'daemon',
agentId: 'codex',
agents: [
{ id: 'claude', available: true },
{ id: 'codex', available: false },
],
}),
).toEqual({
has_available_configure_cli: true,
configure_type: 'local_cli',
configure_availability: 'unavailable',
});
});
});
describe('setConfigureGlobals (web client)', () => {
// Reset the module-level state so other suites do not bleed in.
beforeEach(() => {
setConfigureGlobals(BOOT_DEFAULTS);
});
afterEach(() => {
setConfigureGlobals(BOOT_DEFAULTS);
});
it('stores the latest configure-state for downstream captures', () => {
expect(getConfigureGlobals()).toEqual(BOOT_DEFAULTS);
setConfigureGlobals({
has_available_configure_cli: true,
configure_type: 'local_cli',
configure_availability: 'available',
});
expect(getConfigureGlobals()).toEqual({
has_available_configure_cli: true,
configure_type: 'local_cli',
configure_availability: 'available',
});
});
it('never throws when no PostHog client is initialized', () => {
expect(() =>
setConfigureGlobals({
has_available_configure_cli: true,
configure_type: 'both',
configure_availability: 'available',
}),
).not.toThrow();
});
});