mirror of
https://github.com/nexu-io/open-design.git
synced 2026-06-01 03:14:35 +07:00
feat(skills): teach hyperframes skill the HTML-in-Canvas API (#852)
* feat(skills): teach hyperframes skill the HTML-in-Canvas API Vendored skill predates upstream v0.5.1, which added the drawElementImage HTML-in-Canvas guide and the vfx-iphone-device / vfx-liquid-glass / vfx-portal catalog blocks. Without that reference, agents asked to build 'live HTML on a 3D phone screen' compositions had no idea the API existed and produced renders where the device screen was blank or static. - Add references/html-in-canvas.md adapted from the upstream guide, with an OD-specific note about render-loop re-capture (the most common reason output 'looks dead' inside a generated composition). - Cross-link the new reference from SKILL.md and add triggers for 'html in canvas', 'drawElementImage', 'html shader', and the two most-requested vfx blocks. Daemon render path is unchanged: 'npx hyperframes render' already auto-enables --enable-features=CanvasDrawElement, and we always pull the latest published hyperframes via npx, so no version pinning needs to move. * docs(skills): wait for canvas paint in hyperframes HTML-in-Canvas examples The drawElementImage API only refreshes its element snapshot when the canvas paints. Calling it during initial script evaluation can throw because no snapshot exists yet, and calling it later from outside a paint event silently reads the previous snapshot. On HyperFrames' seek-driven renders that surfaces as a failed or stale first texture. - Drive the basic capture example from canvas.onpaint and kick it off with requestPaint() instead of calling drawElementImage at script eval time. - Rewrite the per-frame re-capture pattern to put drawElementImage inside onpaint and call requestPaint() from the render loop, so each frame sees a fresh snapshot rather than the previous one. - Add a callout explaining the paint-event requirement so agents do not regress to the script-eval-time pattern. Generated-By: looper 0.6.1 (runner=fixer, agent=claude-code) * docs(skills): add vfx-portal trigger to hyperframes skill The hyperframes skill's reference callout already names vfx-portal alongside vfx-iphone-device and vfx-liquid-glass as effects that should auto-load references/html-in-canvas.md, but the triggers list only includes the other two. A prompt like "make a vfx-portal clip" therefore misses the HTML-in-Canvas guidance the new reference adds. - Add "vfx-portal" to triggers so the trigger surface matches the documented entry points. Generated-By: looper 0.6.1 (runner=fixer, agent=claude-code)
This commit is contained in:
Tom Huang
GitHub
parent
32df17b87b
commit
6e473a4f77
2 changed files with 153 additions and 0 deletions
|
|
@ -9,6 +9,12 @@ triggers:
|
|||
- "captions"
|
||||
- "tts video"
|
||||
- "kinetic typography"
|
||||
- "html in canvas"
|
||||
- "drawElementImage"
|
||||
- "html shader"
|
||||
- "vfx-iphone-device"
|
||||
- "vfx-liquid-glass"
|
||||
- "vfx-portal"
|
||||
od:
|
||||
mode: video
|
||||
surface: video
|
||||
|
|
@ -489,5 +495,6 @@ Skip on small edits (fixing a color, adjusting one duration). Run on new composi
|
|||
- **[references/transitions.md](references/transitions.md)** — Scene transitions: crossfades, wipes, reveals, shader transitions. Energy/mood selection, CSS vs WebGL guidance. **Always read for multi-scene compositions** — scenes without transitions feel like jump cuts.
|
||||
- [transitions/catalog.md](references/transitions/catalog.md) — Hard rules, scene template, and routing to per-type implementation code.
|
||||
- Shader transitions are in `@hyperframes/shader-transitions` (`packages/shader-transitions/`) — read package source, not skill files.
|
||||
- **[references/html-in-canvas.md](references/html-in-canvas.md)** — HTML-in-Canvas (`drawElementImage`) for rendering live DOM as WebGL textures: 3D device mockups, shader-warped UIs, liquid glass, portals. Read when the user asks for `vfx-iphone-device`, `vfx-liquid-glass`, `vfx-portal`, or any "HTML mapped onto 3D / shader" effect. The render path auto-enables the Chrome flag, but the texture must be re-captured every frame for animated content — that's the most common cause of "the screen renders dead" output.
|
||||
|
||||
GSAP patterns and effects are in the `/gsap` skill.
|
||||
|
|
|
|||
146
skills/hyperframes/references/html-in-canvas.md
Normal file
146
skills/hyperframes/references/html-in-canvas.md
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
# HTML-in-Canvas
|
||||
|
||||
Render live HTML as WebGL textures — GPU shaders, 3D geometry, and cinematic effects on any DOM content.
|
||||
|
||||
The HTML-in-Canvas API (`drawElementImage`) lets you capture live, rendered DOM elements directly into a canvas at GPU speed. This means you can take any HTML — dashboards, forms, landing pages, app UIs — and render them as textures in WebGL scenes with shaders, 3D transformations, and post-processing effects.
|
||||
|
||||
> **Chrome flag required for live preview only.** The `drawElementImage` API is experimental.
|
||||
>
|
||||
> 1. Open `chrome://flags/#canvas-draw-element` in Chrome or Brave
|
||||
> 2. Set **CanvasDrawElement** to **Enabled**
|
||||
> 3. Restart the browser
|
||||
>
|
||||
> HyperFrames enables this flag automatically during rendering (`--enable-features=CanvasDrawElement`), so rendered videos work without manual setup. The flag is only needed for live preview in the Studio.
|
||||
|
||||
When this skill runs inside Open Design, the daemon shells out to `npx hyperframes render`, which inherits the auto-enable. You do **not** need to add browser flags or pass extra CLI args from the agent.
|
||||
|
||||
## How it works
|
||||
|
||||
1. Place HTML content inside a `<canvas layoutsubtree>` element
|
||||
2. The browser renders the HTML children as normal DOM
|
||||
3. Wait for the canvas to paint, then call `ctx.drawElementImage(element, x, y, w, h)` to capture the rendered pixels
|
||||
4. Use the canvas as a Three.js texture, apply shaders, map to 3D geometry
|
||||
|
||||
> **Always capture from a paint event.** The element snapshot the API draws from is only refreshed when the canvas paints. Calling `drawElementImage` during initial script evaluation can throw because the first snapshot does not exist yet; calling it outside `paint` after that point silently reads the *previous* snapshot. Drive both first-time capture and per-frame updates from `canvas.onpaint`, and use `canvas.requestPaint()` to ask for a fresh snapshot.
|
||||
|
||||
```html
|
||||
<!-- 1. HTML content lives inside the canvas -->
|
||||
<canvas id="capture" layoutsubtree width="1920" height="1080">
|
||||
<div class="my-dashboard">
|
||||
<h1>Revenue: $4.2M</h1>
|
||||
<div class="chart">...</div>
|
||||
</div>
|
||||
</canvas>
|
||||
|
||||
<!-- 2. WebGL canvas for 3D rendering -->
|
||||
<canvas id="theater" width="1920" height="1080"></canvas>
|
||||
```
|
||||
|
||||
```javascript
|
||||
// 3. Capture HTML to canvas — wait for paint so the element snapshot exists
|
||||
var capCanvas = document.getElementById("capture");
|
||||
var ctx = capCanvas.getContext("2d");
|
||||
var texture, material;
|
||||
|
||||
capCanvas.onpaint = function () {
|
||||
ctx.drawElementImage(capCanvas.querySelector(".my-dashboard"), 0, 0, 1920, 1080);
|
||||
if (!texture) {
|
||||
// 4. Use as Three.js texture
|
||||
texture = new THREE.CanvasTexture(capCanvas);
|
||||
material = new THREE.MeshBasicMaterial({ map: texture });
|
||||
} else {
|
||||
texture.needsUpdate = true;
|
||||
}
|
||||
};
|
||||
|
||||
// Kick off the first paint; subsequent re-captures call requestPaint() again
|
||||
capCanvas.requestPaint();
|
||||
```
|
||||
|
||||
## What makes this different
|
||||
|
||||
Traditional approaches like `html2canvas` re-parse and re-render the DOM in JavaScript — they're slow, lossy, and miss CSS features like `backdrop-filter`, complex shadows, and web fonts. The `drawElementImage` API uses the browser's own compositor, so:
|
||||
|
||||
- **Pixel-perfect** — every CSS feature is supported because the browser renders it natively
|
||||
- **GPU-accelerated** — captures at 60fps, fast enough for real-time animation
|
||||
- **Live content** — the HTML can animate, scroll, and change between captures
|
||||
- **Multiple captures simultaneously** — no nesting restrictions; multiple `<canvas layoutsubtree>` elements can capture different content in the same composition
|
||||
|
||||
## Feature detection
|
||||
|
||||
Always feature-detect before using the API. Compositions should fall back gracefully for browsers without the flag enabled. (Render path is always fine — the fallback only matters when a user opens the composition in a browser without `CanvasDrawElement`.)
|
||||
|
||||
```javascript
|
||||
function isSupported() {
|
||||
var tc = document.createElement("canvas");
|
||||
if (!("layoutSubtree" in tc)) return false;
|
||||
tc.setAttribute("layoutsubtree", "");
|
||||
var ctx = tc.getContext("2d");
|
||||
return ctx && typeof ctx.drawElementImage === "function";
|
||||
}
|
||||
|
||||
if (isSupported()) {
|
||||
ctx.drawElementImage(element, 0, 0, w, h);
|
||||
} else {
|
||||
// Fallback: draw text directly on canvas, use static image, etc.
|
||||
}
|
||||
```
|
||||
|
||||
## Re-capturing every frame
|
||||
|
||||
For animated content (scrolling, transitions, counters), drive the capture from the canvas's `paint` event and ask for a fresh snapshot each frame with `requestPaint()`. Calling `drawElementImage` directly from the render loop reads the *previous* paint's snapshot, which on seek-driven HyperFrames renders shows up as a stale or frozen first texture.
|
||||
|
||||
```javascript
|
||||
// Capture runs whenever the canvas paints, so the snapshot is always fresh
|
||||
capCanvas.onpaint = function () {
|
||||
ctx.clearRect(0, 0, W, H);
|
||||
ctx.drawElementImage(htmlElement, 0, 0, W, H);
|
||||
texture.needsUpdate = true;
|
||||
};
|
||||
|
||||
function render() {
|
||||
// Update HTML state
|
||||
scrollContainer.style.transform = "translateY(-" + scrollOffset + "px)";
|
||||
counterEl.textContent = Math.round(currentValue);
|
||||
|
||||
// Schedule a fresh snapshot; the onpaint handler above runs the capture
|
||||
capCanvas.requestPaint();
|
||||
|
||||
// Render 3D scene with updated texture
|
||||
renderer.render(scene, camera);
|
||||
}
|
||||
```
|
||||
|
||||
When a HyperFrames timeline drives the underlying HTML (counter ticks, scroll animation), the render loop must run on every frame the texture is visible — otherwise the WebGL surface freezes on the first capture and the user sees a static screen embedded in your 3D scene. This is the most common reason an HTML-in-Canvas composition "looks dead" after rendering.
|
||||
|
||||
## Catalog blocks
|
||||
|
||||
Install all HTML-in-Canvas blocks at once:
|
||||
|
||||
```bash
|
||||
npx hyperframes add html-in-canvas
|
||||
```
|
||||
|
||||
Or install individually:
|
||||
|
||||
| Block | Description | Install |
|
||||
|-------|-------------|---------|
|
||||
| Liquid Glass | Voronoi glass fracture with parallax reveal | `npx hyperframes add vfx-liquid-glass` |
|
||||
| iPhone & MacBook | Real 3D GLTF devices with live HTML screens | `npx hyperframes add vfx-iphone-device` |
|
||||
| Text Cursor | Dramatic text reveal with chromatic shadows | `npx hyperframes add vfx-text-cursor` |
|
||||
| Portal | Dimension breach with volumetric light | `npx hyperframes add vfx-portal` |
|
||||
| Shatter | HTML shatters into glass fragments | `npx hyperframes add vfx-shatter` |
|
||||
| Magnetic | Magnetic field particle visualization | `npx hyperframes add vfx-magnetic` |
|
||||
| Liquid Background | Organic liquid simulation | `npx hyperframes add vfx-liquid-background` |
|
||||
|
||||
Block reference pages live at `https://hyperframes.heygen.com/catalog/blocks/<name>`.
|
||||
|
||||
## Rendering
|
||||
|
||||
HyperFrames enables the Chrome flag automatically during rendering. No special configuration needed:
|
||||
|
||||
```bash
|
||||
npx hyperframes render --output my-video.mp4
|
||||
```
|
||||
|
||||
For Docker renders, the flag is also enabled automatically inside the container. Inside Open Design, the daemon's `npx hyperframes render` call (`apps/daemon/src/media.ts`) inherits the same default — you don't need to thread anything through.
|
||||
Loading…
Reference in a new issue