Member-only story
Streaming structured data from LLMs is harder than you think
The moment you need both real-time streaming and typed JSON from an LLM, you hit a wall. Every production team building AI-powered systems discovers this: LLMs generate tokens one at a time, and every intermediate state of a JSON response is syntactically invalid. You can stream raw text beautifully, but the instant you need structured data the kind backends and frontends actually consume naive streaming falls apart.
BAML, a domain-specific language for type-safe LLM functions, solves this with a resilient parser that transforms broken partial JSON into typed objects in real time, giving you progressive structured data without sacrificing the UX benefits of streaming.
Why streaming is table stakes for production LLM systems
LLMs are autoregressive; they produce one token at a time, and each token depends on everything before it. A 500-token response from GPT-4 Turbo takes 8–15 seconds to generate fully. Without streaming, your user stares at a spinner for that entire duration. With streaming, the first token arrives in ~0.5 seconds, and content flows progressively from there.
The transport mechanism is straightforward. Nearly every major LLM provider uses Server-Sent Events (SSE) over HTTP. The server pushes data: {JSON} chunks…
