Skip to main content

Introduction

social

This site documents the IaC (infrastructure-as-code) logic for deploying the JoyAI vision-language interaction model onto AWS, driven entirely through Claude Code Agent Skills.

The Vision-Language Interaction Model​

Unlike traditional "turn-based" AI systems that wait for a user to press a button or ask a question before responding, this model is designed to be proactive. It continuously processes a live video stream (such as a webcam, a livestream, or a security feed) and autonomously determines how and when to act in real time.

Every second, the model internally evaluates the visual stream and chooses between three primary signals:

  1. Silence (</silence>): It determines that nothing important is happening and remains quiet.
  2. Response (</response> text): It proactively speaks up or answers based on a visual event it just observed (e.g., commenting on a live sports game, giving step-by-step cooking instructions, or triggering an emergency alert).
  3. Delegate (</delegate> task): If a visual situation requires heavy reasoning or specialized knowledge beyond its inline capability (like a complex math problem or long-horizon reasoning), it hands the subtask off asynchronously to a background agent or a larger language model, allowing the main interaction model to keep watching the stream without freezing up. See Delegation & Screenshot Enhancement for a concrete, working implementation of this signal.

The upstream repository includes the complete deployable system architecture to support this behavior, featuring a WebUI, context-compressing memory systems for long video sessions, and pluggable ASR (Speech-to-Text) and TTS (Text-to-Speech) modules running on standard vLLM. It is built to bridge the gap between passive video analysis and the real-time demands required for embodied intelligence and proactive assistants.

What's Good About This Model for Waybox​

The core differences between running JoyAI-VL-Interaction on Waybox's cloud versus routing visual streams through the Gemini HTTP API come down to structural design, control, and real-time mechanics:

1. True Local State Autonomy (No "Turn-Based" Delay)​

When we communicate with a standard cloud model via an HTTP API, the interaction is traditionally turn-based. We send a frame or video clip payload, the API processes it, returns a single text block, and then the connection closes.

JoyAI runs an always-on, per-second streaming loop on our raw hardware. The model itself decides every single second whether to emit text, execute a background task, or remain entirely silent based on a continuous visual buffer. It acts like an in-room observer rather than a passive remote assistant waiting for an explicit request payload.

2. Predictable Compute Cost at Scale​

Using a cloud API for real-time video streaming requires constant token ingestion. If we feed a high-resolution frame every second to a remote endpoint, we are hitting the API 3,600 times per hour. At scale, this translates into astronomical token fees because we pay for every single input and output token continuously.

With the EC2 architecture, our infrastructure costs are completely capped. We pay a flat rate for the GPU uptime, regardless of whether we process 10 minutes or 10 hours of live, uninterrupted stream video.

3. Zero Data Leakage & Vendor Dependency​

When routing live cameras or audio feeds through a third-party API, sensitive physical operational environments or private user environments must pass through external corporate infrastructure. Running JoyAI means our video streams never leave our virtual private cloud (VPC), ensuring total data privacy. Furthermore, our application is entirely insulated from external API outages, rate limits, or sudden model deprecation cycles.

4. Sub-Second Network Latency​

Streaming audio and video inputs over standard public HTTP connections introduces unavoidable overhead: DNS resolution, TLS handshakes, data serialization, and cloud load-balancer routing. For a system designed to react instantly to a moving object or a spoken word, that extra latency breaks the illusion of immediate presence. By utilizing direct WebRTC pipelines straight into our cloud vLLM system, we eliminate the public internet round-trip bottleneck.