vndangkhoa/open-design
1
0
Fork 0
mirror of https://github.com/nexu-io/open-design.git synced 2026-05-31 19:04:39 +07:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
open-design/e2e/AGENTS.md
Patrick A 32fd5286b5
chore(e2e): improve test framework quality (#2305) 
* chore(e2e): improve test framework quality

- Add lib/timeouts.ts with CI-scaled short/medium/long/xlong constants
- Add lib/playwright/mock-factory.ts to centralise standard localStorage,
  /api/agents, and /api/app-config mock setup; migrate critical-smoke and
  workspace-keyboard-flows to use applyStandardMocks()
- Delete empty lib/shared.ts placeholder
- Replace waitFor({ state: 'detached' }).catch(() => {}) with
  waitFor({ state: 'hidden' }) in all UI tests; 'hidden' resolves
  immediately when the element was never in the DOM, eliminating the
  silent error-swallowing catch
- Remove redundant .catch(() => false) from all isVisible() call sites
  since isVisible() never throws in Playwright
- Convert .waitFor().then(() => true).catch(() => false) guards in
  openDesignFile() to explicit try/catch blocks for clarity
- Simplify sendPrompt() in app.test.ts: replace the 3-attempt manual
  retry loop with a single fill + pressSequentially fallback; the core
  workaround for contenteditable unreliability is preserved but the
  loop structure is gone

* fix(e2e): guard routeMockAgents to GET only

routeMockAgents was intercepting all HTTP methods and returning the mock
fixture, silently swallowing any agent mutation requests. Mirror the
GET-only guard from routeAppConfig so writes fall through to the daemon.

* fix(e2e): address code review findings

- sendPrompt() in app.test.ts, workspace-keyboard-flows.test.ts,
  app-restoration.test.ts: drop fill() (unreliable on contenteditable,
  inputValue() always returns '' for them) and go straight to
  pressSequentially(), which types key-by-key and is authoritative
- Import T from timeouts.ts in app.test.ts and use T.short for the
  input/button waits, making the timeouts module non-dead

* fix(e2e): resolve adversarial review findings

- Revert sendPrompt to fill(): chat-composer-input is a textarea, not
  contenteditable; fill() is atomic and ~60x faster than pressSequentially
- Use T.medium in all waitForLoadingToClear calls: CI workers scale this
  to 20s automatically via the CI env var, eliminating cold-runner flakes
- Add T import to 6 files that needed it for T.medium
- Fix openDesignFile try/catch scope in app-manual-edit: previously the
  catch block only caught waitFor but click/expect errors were also swallowed;
  now only waitFor is inside try, real interaction failures propagate
- Fix regex escaping: .replace('.', '\\.') -> .replace(/\./g, '\\.') in
  app-manual-edit and app-design-files to handle multi-dot filenames
- Migrate entry-chrome-flows.test.ts to applyStandardMocks: it had the
  identical 3-call setup pattern as the factory but was not migrated
- Add GET method guard to project-management-flows app-config route handler,
  matching the pattern used by every other route handler in the suite
- Remove no-op 'as const' from timeouts.ts: Math.ceil returns number,
  not a literal, so the assertion had no effect
- Update e2e/AGENTS.md: remove deleted lib/shared.ts entry, document
  lib/timeouts.ts and lib/playwright/mock-factory.ts

* fix(e2e): scope openDesignFile try/catch to waitFor only

Move click and expect(preview).toBeVisible() outside the catch block so
that a regression in either open path (tab-click or file-list fallback)
fails loudly instead of being silently absorbed. The try now wraps only
the fileTabButton.waitFor existence probe; the subsequent click and final
assertion are unconditional.

---------

Co-authored-by: Patrick A <186436799+eefynet@users.noreply.github.com>
Co-authored-by: Patrick A <259201958+eefynet@users.noreply.github.com>
2026-05-23 00:24:32 +08:00

5.5 KiB
Raw Blame History

e2e/AGENTS.md

Follow the root AGENTS.md first. This package owns user-level end-to-end smoke tests and Playwright UI automation only.

For the current coverage posture, recent hardening work, grouped-run status, and known intentional gaps, see docs/testing/e2e-coverage/status.md.

Directory layout

  • specs/: highest-ROI, long-running core business capability regressions suitable for PR or release gating. Each spec should describe one nearly orthogonal product capability chain, such as main dialog generation, Pet, Orbit, or packaged runtime. Keep this layer small and expand it only when a core capability deserves always-on signal.
  • tests/: broader user-level end-to-end coverage and local hotspot checks that intentionally span app/package/resource boundaries. Prefer adding tests here when a repeated or high-risk local capability naturally falls out of a core spec. Do not build a speculative coverage matrix before the core spec needs it.
  • ui/: flat Playwright UI automation test files only. Keep helpers, resources, and non-Playwright harnesses out of this directory.
  • resources/: declarative resources for e2e suites, such as Playwright UI scenario lists.
  • lib/fake-agents.ts: shared fake local agent CLI harness used by UI and pure-inspect daemon specs.
  • lib/timeouts.ts: CI-scaled timeout constants (T.short, T.medium, T.long, T.xlong). Import as { T } from @/timeouts. Use these instead of hardcoded millisecond values in UI tests.
  • lib/playwright/mock-factory.ts: shared Playwright mock helpers. applyStandardMocks(page) seeds localStorage and intercepts /api/agents and /api/app-config with standard daemon/mock-agent fixtures. Use in beforeEach for tests that do not need a custom agent or protocol setup.
  • lib/vitest/: Vitest-specific atomic helpers only. Helpers describe actions such as namespace lifecycle, mock servers, HTTP calls, tools-dev commands, inspect, logs, and reports; they should not hide core business scenario decisions.
  • lib/vitest/report.ts: the report boundary. Specs save curated output through report.save(<relpath>, <blob>) or report.json(<relpath>, value); release workflows should consume only the final report path, not its internal file layout.
  • createSmokeSuite(...).with.*: suite-owned lifecycle composition. Prefer this shape for namespace-bound resources such as suite.with.toolsDev(...) so specs keep business workflow code in the foreground.
  • lib/playwright/: Playwright-specific fixtures, resource accessors, route helpers, and UI actions.
  • scripts/playwright.ts: Playwright auxiliary subcommands such as artifact cleanup; it must not wrap playwright test.

Spec and test model

  • Start from specs/: define orthogonal long-form core capabilities first, then let supporting tests/ and lib/ grow from those chains.
  • specs/ should read as business/system workflows, for example dialog/main.spec.ts, orbit/run.spec.ts, or pet/main.spec.ts.
  • tests/ should pin reusable local hotspots, such as tools-dev/inspect.test.ts, provider mocks, report lifecycle, artifact file shape, or namespace cleanup.
  • High-confidence infrastructure checks may be added to tests/ before a full core spec exists, but most tests should be extracted only after a spec proves the local hotspot matters.
  • Treat tests/ as maintainable support material, not permanent coverage inventory. Merge, split, shrink, or delete tests as product capabilities evolve.
  • Keep new non-UI e2e smoke chains pure inspect by default. Do not use Playwright for these chains; use daemon/web APIs, sidecar IPC, tools-dev/tools-pack inspect, logs, reports, and screenshots when available.
  • External service dependencies must use temporary server-level mocks. Do not rely on real API keys, real provider accounts, or UI-level route patching for core e2e smoke.
  • Every atomic suite must run in an isolated namespace. Successful suites should keep only curated reports and high-value artifacts, then clean process/runtime scratch. Failed suites should preserve runtime scratch, logs, mock requests, screenshots, and report pointers for diagnosis.

Naming and tools

  • specs/ files must be *.spec.ts; tests/ files must be *.test.ts.
  • Prefer directory hierarchy over long file names. Basenames should normally be three words or fewer, such as main.spec.ts, run.spec.ts, inspect.test.ts, or report.test.ts.
  • ui/ files must be flat *.test.ts Playwright tests. Do not add subdirectories, TSX, Vitest, jsdom, Testing Library, or React harness tests under ui/.
  • E2E Vitest tests use Node APIs; do not add JSX/TSX, jsdom, or browser-component tests under specs/ or tests/.
  • Web component/runtime tests belong in apps/web/tests/, not e2e/ui/.
  • E2E tests may validate cross-app/resource consistency, but must not treat one app's private implementation as a shared helper for another app. Keep test-only helpers local to e2e/lib/ or promote reusable logic to a pure package such as packages/contracts.
  • E2E imports may use @/* for lib/*; keep this alias local to the e2e package.

Commands

Run commands from this directory:

pnpm test specs/mac.spec.ts
pnpm test tests/tools-dev/inspect.test.ts
pnpm test specs
pnpm test tests
pnpm typecheck
pnpm exec tsx scripts/playwright.ts clean
pnpm exec playwright test -c playwright.config.ts --list
pnpm exec playwright test -c playwright.config.ts

Use a specific file path when validating a single case. Do not add root e2e aliases or extra package scripts for individual cases.