Skip to main content

Delegation & Screenshot Enhancement

The interaction model can hand hard subtasks to a background agent (the third signal alongside silence/response — see Introduction). Two skills turn that into a concrete capability: watch for a described moment in the video, screenshot it, and polish it with an image model.

1. Install and Start the Background Agent​

Set up the background agent

This invokes the background-agent-setup skill — a code-execution agent (the system codex CLI) that the model can delegate to, including handing it actual frames from the stream.

:::info Prerequisites SSH access to the instance, the WebUI and webinfer services already running, an OpenAI API key (there's no way to fabricate this — the skill will ask), and ~300MB free disk for the Codex CLI binary. :::

It installs the Codex CLI, authenticates it (asks how — API key, ChatGPT device login, or copying existing local credentials), and starts the service. No WebUI restart is needed afterward — new sessions pick up delegation automatically.

2. Have the Model Watch for Something and Act on It​

Two equivalent ways to do this:

Ask the agent to set it up for you:

Whenever you see <some event>, capture and enhance a screenshot of it

This invokes the event-screenshot-enhance skill, which sends a one-shot steering prompt over the WebUI's /ws connection — the exact prompt below, with <some event> substituted in.

Or paste the underlying prompt directly into the WebUI's own "Message the VLM..." box yourself (reload the page first — see the reliability note below), substituting the event you want watched for:

Whenever you see <some event>, output exactly: a short note that you're delegating, then delegate by asking
the background agent to (a) save and describe a screenshot of that exact moment, and (b) use the OpenAI
images API (gpt-image-1) to create a polished, higher-resolution version of it, then report the full
ABSOLUTE file paths (starting with /) of both the original and polished images in your final summary. Do
not describe the scene yourself in your response - keep your own response to one short sentence and let the
delegation handle the rest.

Either way, this instructs the model to delegate once it sees the described event, and instructs the background agent to locate the exact frame, save it, and call OpenAI's gpt-image-1 image-edit API to produce a polished, higher-resolution version, reporting back full absolute file paths so the WebUI auto-links them — viewable directly by clicking the link in the resulting chat message.

:::tip Reliability Reliability depends on starting from a fresh WebUI session (reload the page) right before sending the prompt — once a session has seen the model narrate about delegating without actually doing it, it tends to keep repeating that habit. A visual event that stays on screen for a few seconds can also trigger several redundant delegations in a row, each a real Codex + gpt-image-1 API call — worth moving past the scene (or pausing the feed) once a good result comes back, rather than letting it keep re-firing. :::

:::caution Quality caveat gpt-image-1's edit endpoint is generative, not a pixel-preserving upscaler — fine details like on-screen text can shift slightly. Swap in a true super-resolution model instead if exact fidelity matters more than polish. :::

How It Works, and Running It Without Opening the WebUI at All​

Whichever of the two prompt options above triggers it, the underlying mechanism is the same: a JSON message {"type": "update_prompt", "prompt": "..."} sent over the WebUI's /ws?session_id=... WebSocket — the identical message the browser's own "Message the VLM..." box sends. That prompt becomes standing context steering the model's per-second silence/response/delegate decision from that point on, not a one-off question. Sessions live entirely server-side, keyed only by session_id, so:

  • If the agent applies this against a session your browser already has open, the prompt update itself won't visibly appear in the chat (the frontend only logs it to the browser's dev console) — but every event after that — the model's responses, the delegation-in-progress notice, and the final polished-image link — broadcasts to every client connected to that session, so it all still shows up live in your browser exactly as if you'd typed the prompt yourself.
  • A browser isn't required at all, though. The agent can create a brand-new session directly by calling POST /api/rtsp/start with a fresh session ID (no page load involved), send the steering prompt over a WebSocket it opens itself, watch the server log for Background delegation queued / Background result ready on that session, and download the resulting images straight off the instance once they land — the entire capture-and-enhance pipeline runs headless, end to end, without the WebUI ever being open in a browser.