OpenUI: Generative UI widgets (experimental)

Experimental. This guide describes a freshly landed plugin pair that depends on @openuidev/* packages still on a 0.x release cadence. APIs may shift; pin versions and re-evaluate when @openuidev ships 1.0.

uSketch can host AI-generated UI widgets — a pricing card, a contact form, a stat dashboard — as first-class shapes on the canvas. Two plugins from this repo cover the round-trip:

@edv4h/usketch-plugin-shape-openui — defines the openui shape and renders OpenUI Lang snippets via <Renderer> from @openuidev/react-lang.
@edv4h/usketch-plugin-tool-openui — registers a toolbar tool, a side-panel prompt UI, a “Make Real” button on selection, and the LLM provider plumbing (OpenAI / OpenAI-compatible).

The flow is tldraw’s “Make Real” pattern but with OpenUI Lang instead of raw Tailwind HTML: outputs are validated against a Zod-schema-defined component library before they hit the DOM, so a malicious LLM cannot inject arbitrary script tags.

Setup

Both plugins ship with the monorepo and are wired into apps/web for cloud boards (/boards/...); local boards don’t load them. For a standalone host:

pnpm add @edv4h/usketch-plugin-shape-openui @edv4h/usketch-plugin-tool-openui

In apps/web, OpenUI LLM calls go through the server-side proxy at /api/ai/openui (provided by @edv4h/usketch-plugin-server-ai). The OpenAI API key lives only in the server’s Workers Secret store — it never ships in the browser bundle.

The only build-time env var you can optionally set:

VITE_OPENUI_MODEL — pick a default model name. Defaults to gpt-4o.

Production safety: Do not set VITE_OPENAI_API_KEY, VITE_OPENUI_BASE_URL, or VITE_OPENUI_PROVIDER in Cloudflare Pages build env vars. They are no longer read by the app, but historical configs may still leak via Vite’s import.meta.env if you re-introduce them. The API key must only live in the server’s OPENAI_API_KEY Workers Secret.

Deploying

Server (Cloudflare Workers): make sure OPENAI_API_KEY is set as a secret (wrangler secret put OPENAI_API_KEY — already required by /api/ai/complete). The OpenUI route reuses the same secret.
Web (Cloudflare Pages): set VITE_API_URL to your worker URL. Optionally set VITE_OPENUI_MODEL. Deploy the server before the web app on cutover — old web builds pointing at a worker without the route will see 404s.

Wiring it into createApp

import { createApp } from "@edv4h/usketch-core";
import { createSidePanelPlugin } from "@edv4h/usketch-plugin-side-panel";
import { openUIShapePlugin } from "@edv4h/usketch-plugin-shape-openui";
import {
	createOpenUIToolPlugin,
	createServerProxyProvider,
} from "@edv4h/usketch-plugin-tool-openui";

const app = await createApp({
	store,
	plugins: [
		// Side panel must come BEFORE the tool plugin so its tab registration is received.
		createSidePanelPlugin(),
		openUIShapePlugin,
		createOpenUIToolPlugin({
			provider: createServerProxyProvider({
				apiUrl: import.meta.env.VITE_API_URL,
				boardId,
				// extraHeaders: { "X-User-Id": "dev-user" }, // dev-mode bypass only
			}),
		}),
	],
});

Usage

Click the ✨ OpenUI tool (or press U) — the side panel opens to the OpenUI tab.
Type a description, e.g. “a pricing card with three tiers and CTA buttons”.
Hit Generate (or ⌘+Enter). The plugin streams LLM output into a buffer, then places a new openui shape at the viewport center once the response completes. Zod-schema validation against the library happens at render time inside <Renderer> — invalid components are dropped from the rendered output (and surfaced as Renderer error: in the console), but the shape itself is still created.

The status row reports streaming progress; the Cancel button aborts in-flight requests cleanly.

Make Real

Select any shapes (a freehand sketch, a stack of stickies, a wireframe drawn with rectangles), then click ✨ Make Real in the floating button above the selection. The plugin:

Snapshots the selection as a PNG via exportCanvas from @edv4h/usketch-plugin-export.
Sends the image plus a “build a real UI matching this sketch” prompt to the provider.
Places the result to the right of the source bounds.

@edv4h/usketch-plugin-export is a hard runtime dependency of the tool plugin (the package is listed under dependencies and exportCanvas is imported directly), so make sure your host installs it. The plugin does not gracefully fall back if you skip it — the Make Real button will throw when clicked, and the error surfaces via the standard ai:status channel as a side-panel error banner.

Customizing the library

The library defines what components the LLM is allowed to emit. The default library ships 12 generic primitives (Stack, Heading, Text, Card, Button, Input, Badge, Avatar, Image, List, Row, Spacer). Pass a custom one if you need branded components:

import { createLibrary, defineComponent } from "@openuidev/react-lang";
import { z } from "zod/v4";

const BrandHero = defineComponent({
	name: "BrandHero",
	description: "Hero banner with the Acme brand colors.",
	props: z.object({ title: z.string(), subtitle: z.string().default("") }),
	component: ({ props }) => <header className="acme-hero">…</header>,
});

const library = createLibrary({ components: [BrandHero /* + your other components */] });

createOpenUIToolPlugin({ provider, library, libraryId: "acme-1" });

The system prompt is regenerated from the library on every request via library.prompt(...), so the LLM is always told the exact set of components and their Zod-typed props.

Provider abstraction

import {
	createServerProxyProvider,
	createOpenAIProvider,
	createOpenAICompatibleProvider,
} from "@edv4h/usketch-plugin-tool-openui";

// Production-safe: route through your own server proxy (recommended).
createServerProxyProvider({
	apiUrl: "https://api.usketch.app",
	boardId: "board-abc",
});

// Third-party hosts that already have a server-side proxy can talk to OpenAI
// directly — but never bake a real API key into a browser bundle:
createOpenAIProvider({ apiKey: process.env.OPENAI_API_KEY! });

// Anything OpenAI-compatible (Ollama, vLLM, LiteLLM, Azure OpenAI, …):
createOpenAICompatibleProvider({
	baseURL: "http://localhost:11434/v1",  // Ollama
	defaultModel: "llama3.2:vision",
});

createOpenAIProvider and createOpenAICompatibleProvider remain exported for third-party hosts that have their own backend; in this repo’s apps/web, both have been replaced by createServerProxyProvider.

Anthropic is not directly supported in v1 — server-side proxying with a normalized SSE format is the planned approach. Track the follow-up issue if you need it.

Security model

The <Renderer> from @openuidev/react-lang validates every parsed statement against the library’s Zod schemas. Components or props the library doesn’t declare are rejected before they render — there is no dangerouslySetInnerHTML and no script execution surface.
The default createServerProxyProvider keeps the OpenAI API key out of the browser bundle entirely. Authentication relies on the session cookie (credentials: "include"); the dev-mode X-User-Id header is a bypass that should never be enabled in production.
The server route enforces board access control when a boardId is supplied (membership or isPublic).
Vision requests for Make Real send the canvas snapshot as a base64 data URL up to 10 MB; the server route validates the payload size before forwarding. For sensitive boards, deploy a middleware that strips identifiers before the proxy forwards to OpenAI.

Troubleshooting

Symptom	Likely cause
Shape mounts as “OpenUI library not configured” fallback	`createOpenUIToolPlugin` (or another caller of `setOpenUILibrary`) is not registered in `createApp({ plugins })`. Plugin ordering doesn’t matter — `createApp` finishes every plugin’s `setup()` before the canvas paints — but if no caller wires up a library, the shape renders the fallback.
`OpenUI provider 401:`	The session cookie is missing or expired. Sign in again. In dev, set the `X-User-Id` header through `extraHeaders`.
`OpenUI provider 404:`	The user is not a member of the supplied `boardId`, and the board is private. Either omit `boardId` or share the board.
`OpenUI provider 502:`	The server proxy was unable to reach OpenAI. Check the worker logs and that `OPENAI_API_KEY` is set as a Workers Secret.
`Empty response from model`	The LLM returned only fences / prose. The plugin strips common artifacts; if it still triggers, refine the prompt or pick a stronger model.
The shape renders empty even after `done`	The LLM emitted a component name the library doesn’t define. Open the browser console — `Renderer error:` logs the rejected statement.

Limitations

Static widgets only. Action() (OpenUI Lang’s event hook) is not wired through — buttons render but don’t fire real events. Follow-up tracks deeper integration.
No streaming progressive render in the canvas. The plugin buffers the entire response before placing the shape; partial UI is reported only in the side panel status. tldraw “Make Real” makes the same trade-off.
One library per app instance. The shape plugin holds a single active Library reference. Hosts that want multiple libraries should file an issue describing the use case.