This is the second article in the “Agent Infrastructure and Interaction Design” series. The first article, Interface Standards as Entropy Reduction, started from stateful containers and discussed skill architecture, environment isolation, and why interface standards are the most important design work of the agent era. This article focuses on generative UI — what it really is, what new problems it introduces, and what role it should play in the overall agent ecosystem.
Future articles will provide technical analysis and hands-on explorations of specific generative UI frameworks like MCP Apps and Google A2UI.
Tokens Aren’t Enough
I recently came across an analogy that felt remarkably accurate: today’s tokens are like mobile data in 2009.
In 2009, mobile internet was just getting started. Everyone was on 2G/3G networks, scraping by with a few hundred megabytes of data per month. Every product design decision revolved around a single constraint — how to deliver as much information as possible within limited bandwidth. Images had to be compressed, video was out of the question, and web pages needed mobile-specific layouts. Bandwidth was the scarce resource, and the entire industry’s innovation was about doing the best within that constraint.
Today’s AI landscape is at a similar stage. Tokens are the core scarce resource — context windows have upper limits, every inference call has a cost, and both input and output are constrained by token counts. Natural language, as the near-exclusive interaction channel between humans and AI, is actually quite inefficient: you need a lot of words to describe a problem, and the AI needs a lot of words to explain its analysis.
But looking back at mobile internet history, bandwidth constraints were eventually dissolved by technological progress — after 4G, 5G, and widespread WiFi, bandwidth stopped being a bottleneck, and product forms evolved from text-and-images to short video, livestreaming, and real-time interaction. When constraints disappear, the needs they suppressed explode.
Token constraints are also receding fast. Context windows have expanded from 4K to million-level in just two years, and inference costs continue to drop. When tokens are no longer the bottleneck, the interaction between AI and humans will break beyond pure text.
This is the backdrop against which generative UI emerges. But before discussing its technical form, we need to answer a more fundamental question: when AI can interact with people in richer ways, what exactly are users consuming?
From Consuming Information to Consuming Intelligence
In the traditional App era, users consumed information. Weather data, news, products, social feeds exist in the world, and apps helped you find, organize, and present them. Apps were information couriers and curators. The competition between apps was about information acquisition, organization, and presentation quality. The entire UI design discipline centered on this question — how to display information clearly, beautifully, and efficiently.
In the Agent era, information acquisition and organization are taken over by AI. Users no longer need to browse, search, or filter on their own. What users are truly consuming is not information — information is raw material — but AI’s ability to understand, judge, and act on information.
Users are consuming intelligence itself.
This shift has fundamental implications for interfaces. If users consume information, the interface’s core task is presentation. If users consume intelligence, the core task changes — the interface needs to help users perceive, understand, verify, and trust AI’s intelligent output. You’re not just looking at a data chart; you’re evaluating whether the agent’s understanding of the data is sound. You’re not just reading an analysis; you’re judging whether the agent’s reasoning is worth adopting.
When you consume information, correctness is relatively easy to verify — is the data accurate, are the facts right? When you consume intelligence, you face probabilistic output with interpretive latitude, making verification far more difficult.
Pure text conversation as the sole interaction channel is insufficient for the task of “helping users verify intelligence.” We need a second channel. This is the deeper motivation behind generative UI.
What Is Generative UI
In simple terms, generative UI means AI dynamically generates interface elements during conversation, rather than having users navigate to pre-designed pages.
Traditional AI conversation products can only output text. You ask an agent a question, it replies with markdown. But text has limited expressiveness — if you ask an agent to analyze a dataset, it can only describe in words that there’s “an upward trend with an anomaly in February.” Generative UI lets the agent embed a chart, a map, a clickable option card, or even a small interactive application directly in its response.
Several products are already exploring this direction. Vercel’s AI SDK supports rendering React components within chat streams; Claude’s artifacts let AI generate complete interactive applications; Google’s A2UI protocol and MCP Apps are also defining standard ways for agents to output rich media.
These implementations differ in approach, but share a core idea: AI’s output is no longer limited to text streams — it can include structured, interactive visual elements. Future articles will analyze these frameworks individually. This article addresses a more fundamental question: why we need this, and what role it should play in the agent ecosystem.
Why We Need a Second Information Channel
With the basic form of generative UI established, we can understand its value at a more abstract level.
The most primitive form of human-AI interaction is natural language. You say something, AI replies with text, you read and understand. This channel is one-dimensional, sequential, decoded linearly by the human brain. Generative UI opens a fundamentally different information channel — instead of describing with language, AI encodes its understanding into visual structures: charts, interactive components, spatial layouts.
This channel has a longer pipeline than natural language:
Every additional step in the pipeline can introduce distortion. But its value comes from three dimensions:
First, bandwidth. The visual channel’s information density far exceeds text. The amount of information a single chart can convey in an instant might require thousands of words to explain. Moreover, visual information is received holistically and explored on-demand, rather than scanned linearly.
Second, precision. This may be more important than bandwidth. Natural language is inherently ambiguous. When an agent says “February sales dropped significantly” — what does “significantly” mean? 5%? 50%? Compared to what baseline? This information is implicit and interpretable in natural language. Two people reading the same sentence might construct completely different magnitudes in their minds. But when this information is rendered as a chart, there must be a precise data model behind it — specific values, well-defined axes, definite time ranges. This data model is unambiguous. The real advantage of generative UI isn’t that it “looks nice” — it forces AI to convert fuzzy understanding into precise data structures. Natural language lets AI be vague — it can say “roughly,” “possibly,” “significantly.” But when AI needs to generate a UI component, it must output structured data. If AI’s understanding of something is fuzzy, generating a data model will expose that.
Third, verifiability. Precisely because there’s a precise data model, users have an anchor point for verifying AI output. You can point to a specific number on a chart and ask “is this value correct,” but you can hardly ask the same question about a phrase like “significant decline.” In the context of “consuming intelligence,” this is especially critical — visualization lets users independently judge whether the agent’s conclusions are reasonable, rather than merely choosing to believe or not.
So generative UI is fundamentally a bandwidth-for-fidelity trade-off. Natural language is narrow but reliable; visual is wide but has a longer pipeline with higher distortion risk. In the context of “consuming intelligence,” this trade-off is worthwhile — the high-bandwidth channel gives users the ability to verify intelligence, and precise data models give that verification an anchor.
But there’s an honest question to face: is generative UI the only way to achieve these values?
GenUI’s Competition
In fact, there are at least three different approaches to making interfaces carry intelligent output in the agent era:
Approach One: Traditional GUI + Agent Backend. The interface is still pre-designed by designers; user interaction is identical to today’s SaaS. The backend logic simply switches from rule engines to agents. Users might not even perceive AI’s presence — they click “Generate Report,” an agent runs behind the scenes, but the frontend is just a button and a results page. Notion AI and Figma AI take this route. The advantage is high certainty — users are familiar with it, and it’s thoroughly designed and tested. The disadvantage is low flexibility — the interface can only present scenarios designers anticipated, and the agent’s capabilities are boxed in by GUI boundaries.
Approach Two: Data Model Driven, UI Decoupled from Dialog. The agent doesn’t generate interfaces directly; instead, it operates an external structured data model through protocols like MCP. The UI layer independently watches for data model changes and decides how to render. Dialog stays dialog, interface stays interface, connected indirectly through a shared data model. The advantage is clean architecture with inherent data precision. The disadvantage is the need for predefined data schemas, and the dialog experience and interface experience are separated.
Approach Three: Generative UI. The agent dynamically generates interface components within the dialog flow. The interface is part of the conversation — most flexible, most natural, but also least controllable.
These three approaches aren’t mutually exclusive. Within the same product, high-frequency deterministic operations take Approach One, data requiring persistent collaboration takes Approach Two, and ad-hoc exploratory presentations take Approach Three.
But this raises a pointed question: if Approaches One and Two already cover most scenarios, what territory does generative UI actually own exclusively?
GenUI’s True Territory
Before answering this question, notice a trend that’s already underway: the relationship between workspace and dialog flow is inverting.
Cursor’s model puts the workspace front-and-center, with the chat box as auxiliary. You stare at the code editor while the agent assists from a sidebar.
But Claude Code and Claude Cowork take a different path — the dialog flow is primary, and the workspace recedes to the background. You describe intent, the agent executes, and execution results come back to you as part of the dialog flow. Your attention focus is no longer the code editor or data panel, but the conversation itself.
If the latter becomes the dominant interaction mode for more and more scenarios, then the dialog flow needs to carry far more than pure text. When the agent says “I modified three files,” you need to see the diff; when it says “analysis results below,” you need to see a chart; when it says “here are two options,” you need to see a comparison view. These things must appear in the dialog flow, because that’s where your attention is.
In this mode, agent interaction naturally splits into three layers:
Serious, certainty-demanding work scenarios. Even when the dialog flow becomes the primary entry point, heavy operations (writing code, designing, editing documents) still need professional workspaces. But entry and orchestration originate from the dialog flow.
Rich media interaction embedded in dialog. This is genUI’s home. You’re discussing where to eat with the agent, and it embeds a map marking several recommended restaurants in the conversation; you’re discussing data trends, and it embeds an interactive chart; you’re planning a trip, and it embeds a schedule card.
Natural language handles most of the long tail. “Summarize this document for me,” “What does this code mean” — a text reply is sufficient; no interface generation needed.
This means genUI’s positioning isn’t “AI-generated applications” but “rich media embedding within dialog flows.” Its granularity is component-level, not application-level. Its lifecycle is within the dialog context, not persistent. Its interaction depth is lightweight — look, click, select — not complex workflows.
Redefining Generative UI
With positioning clarified, genUI’s meaning itself needs reexamination.
A common misconception is that generative UI means “AI generates a complete interface component from scratch.” But looking at actual scenarios, much UI presentation is bound to the tools themselves — map components come from Maps MCP, drawing from Excalidraw, code diffs from code tools. These UIs aren’t dynamically generated by the agent; they come with the tools. The agent merely decides which tool to call.
So what “generative” decisions is the agent actually making? It’s deciding at this point in the dialog flow, how to present information — which tool to invoke, which component to choose, what data to fill, how to compose. What’s generated is the presentation strategy, not UI code.
Understood this way, the entire genUI implementation is actually a spectrum:
The left end is specialized interactive tools. Each tool has its own complete data model, interaction paradigm, and rendering logic. AI only needs to know when to call them and what data to pass. Their advantage is high certainty and high interaction quality. Maps MCP and Excalidraw MCP are this type.
The right end is the general canvas. AI can freely compose and render anything on it. Claude’s artifacts are the prototype — AI generates a React component on the fly, can draw anything, but it’s different every time with unpredictable quality.
The middle is parameterized components. They have fixed interaction paradigms, but content and configuration are flexible. Chart libraries, form generators, comparison cards. AI selects templates, fills data, adjusts parameters. Not generated from scratch, but not fully preset either.
Most real scenarios are covered by the left end and the middle. The general canvas handles the long tail. And there’s a natural evolutionary path — things migrate from right to left. Initially, a presentation need can only be freely rendered on the canvas. When that need recurs, it gets abstracted into a parameterized component. When it further stabilizes, it solidifies into a specialized tool. Canvas is the experimental ground; specialized tools are the end state.
Therefore, the core of generative UI isn’t “generating UI” — it’s the agent’s ability to make presentation decisions across this spectrum based on context. Calling a specialized tool is genUI; configuring a parameterized component is genUI; freely rendering on canvas is also genUI. “Generative” describes the decision process, not the production process.
Skill-to-Dialog Communication
With genUI’s positioning and implementation spectrum established, an infrastructure question surfaces: how does a skill’s execution process interact with the dialog flow?
In the previous article, we discussed the architecture of skills running in serverless containers. The current CLI model is unidirectional — the agent calls a command, the command runs and returns a result. But if a skill needs to embed rich media components in the dialog flow and respond to user actions on those components, it needs a bidirectional, structured message channel with the dialog flow.
The message types this channel needs to carry far exceed traditional stdin/stdout:
In skill code, it might look like this:
from skill_sdk import dialog
# Push an interactive chart to the dialog flow
dialog.emit_surface(
component="chart",
data={"x": dates, "y": values},
interactive=True
)
# Wait for user action
action = dialog.wait_for_interaction()
# Continue processing based on user action
if action.type == "click":
dialog.emit_surface(
component="detail_card",
data=get_details(action.point)
)Under this model, what previously existed as the standalone concept of “interactive MCP” is no longer special — it’s just an ordinary skill that happens to emit interactive surface components and waits for user interaction events. Maps MCP isn’t a special tool protocol; it’s a skill whose surface is a map component, and whose awaited interactions are user actions on the map.
Skills, CLI, interactive tools, and genUI are unified under this model.
Half-Duplex and Operation Intent
But there’s a fundamental architectural issue: the generated surface is both the agent’s expression and the user’s actionable space. If both directions are active simultaneously, conflicts arise — the user is dragging a filter on a chart while the agent simultaneously wants to update the chart with a trend line. Who takes priority?
Fortunately, dialog flow itself provides a natural answer: dialog is inherently half-duplex. You speak, then I speak. GenUI, as an embedding within the dialog flow, naturally inherits this rhythm. The agent pushes a surface, control passes to the user. The user finishes interacting or speaks, and it’s the agent’s turn to respond.
At any given moment, only one party is “speaking” — no concurrent conflicts.
But there’s a subtle question: which user actions on the surface should be immediately sent to the agent, and which are the user exploring on their own?
When a user zooms on a map, hovers over chart values, or scrolls through a table — these are exploratory. The user is still thinking and doesn’t want every action to trigger an agent response. But when a user clicks “Choose A” on a decision card or selects an option in an ask_user_input — these are explicit decisions that should be transmitted immediately.
If all actions are immediately sent to the agent, the user goes crazy. If no actions are sent, the agent is blind.
The reasonable model is: local by default, snapshot attached when speaking.
Most actions on the surface are local — the user views and explores independently; the agent doesn’t intervene. Only a few explicit decision actions (buttons declared as emit_immediately) are transmitted back immediately.
When the user actively types in the chat input, the surface’s current state — what range the user filtered to, which data points are selected, what view they’re on — is automatically attached as a snapshot with the user’s message. The agent doesn’t need to track every user action, but when the user decides to “speak,” the agent can see what the user was looking at and cared about at that moment.
It’s like human collaboration: your colleague spent half an hour browsing data on their own computer — you don’t need to know the browsing process. But when they talk to you, they turn their screen around to show you.
Three-Party Alignment
The above covers genUI’s positioning and infrastructure. Now let’s discuss the core design challenge it introduces.
Traditional GUI design only needs to align two parties: human and system. A designer draws a button; the system knows what clicking it triggers; the human knows what clicking it means. Both sides understanding it the same way — that’s good design.
The agent era adds a third party. When surfaces enter the dialog flow, the information producer (agent), renderer (system), and consumer (human) become three parties, each potentially having different “understandings” of the same surface:
A human sees a chart and understands “sales are declining, February was worst.”
The agent holds the underlying data and analytical logic from generating that chart — its understanding might be “sales are declining, but after seasonal adjustment they’re actually flat.”
The system sees a set of JSON + rendering instructions — it understands no semantics, just faithfully turns data into pixels.
This inconsistency is implicit — everyone is looking at the same thing, but each has a different mental model, with no one aware of the divergence.
However, the data model precision discussed earlier actually provides an anchor point for three-party alignment. If there’s a precise structured data model as an intermediate layer — the system renders from it, the agent generates it, the human understands from its visual projection — the alignment anchor is no longer ambiguous text but precise data structures.
In interactive surfaces, alignment has another dimension. When a user drags a filter on a chart, the operation’s semantics differ for each party. The user thinks “I’m looking at February’s data,” the system executes filter where month=2, and the agent may not know the user did this. The “snapshot attachment” mechanism discussed earlier partially solves this — when the user speaks, the agent sees the surface’s current state — but this is only an approximate alignment, not perfect synchronization.
A surface isn’t just a presentation layer — it’s the medium for shared cognition among human, agent, and system. In the context of “consuming intelligence,” if there’s an implicit gap between what the user understands and what the agent actually understood, the user may make decisions on a flawed foundation without realizing it.
Dual-Channel Cross-Verification
How do we mitigate three-party alignment risks?
Natural language dialog has a built-in fallback mechanism — you said something I didn’t understand, I say “what do you mean,” you rephrase. This repair loop is built into dialog structure.
Generative UI lacks this loop. Once an interface is generated, it just sits there — it won’t proactively ask “did you understand?” If information gets distorted in the pipeline — rendered incorrectly, misread by the user — there’s no mechanism to detect it.
So a key design principle is: genUI should never exist independently from the natural language channel.
Every visual output should have a natural language “shadow” — the agent describes in text “what I generated and what I’m trying to express.”
It serves as alignment verification: if the user understands chart A but the agent’s text description says B, the divergence is immediately visible. Distortion shifts from implicit to explicit.
It serves as a fallback channel: when the visual channel fails — rendering error, user can’t read it, device doesn’t support it — the natural language channel remains, and the user can at least understand what the agent was trying to convey through text.
In the previous article, we proposed the skill surface layering model. Under this framework, Level 0 natural language narration isn’t just “a degraded experience for incapable clients” — it’s the semantic anchor of the entire genUI system:
For skill specification design, this means Level 0 isn’t optional enhancement but a mandatory requirement: any skill that provides Level 1 or Level 2 surfaces must simultaneously provide a Level 0 semantic description, and this description must be sufficient to independently convey the core information.
Affordance Reliability
There’s another common concern about genUI: UI elements are dynamic, so users can’t form operational habits. Each time a new component appears, users might not know how to interact with it.
This concern is real but overestimated. Google’s rich results already validate this pattern at scale — search for flights and a flight card appears, search for recipes and step-by-step rich results appear, search for math formulas and a calculator appears. The UI is different every time, but nobody complains about usability. Because each component’s affordance is self-explanatory.
Human understanding of UI doesn’t depend on “I’ve seen this exact interface before” but on visual language consistency. As long as components follow universal design conventions — buttons look like buttons, input fields look like input fields — users can understand them.
What actually causes problems isn’t component variability, but unpredictable interaction patterns — something looks like a button but triggers an irreversible operation, something looks like an information display card but is actually editable.
So specification constraints shouldn’t be placed on “limiting variability” but on ensuring affordance reliability. Concretely for spec design, UI components generated by skills need to declare their interaction types — display only, clickable, editable, destructive. The client uniformly handles visuals and behavior based on declarations. Skill developers don’t need to design button styles themselves; they just annotate “this operation is destructive,” and the client automatically renders it as a red button with confirmation.
This shifts design consistency responsibility from skill developers to the client runtime. Just as HTML declares “this is a button” and the browser is responsible for making it look and behave like a button.
From Accessibility to Agent Accessibility
Reviewing the above discussion — separating semantics from presentation, multi-channel reachability, progressive enhancement, ensuring predictability on dynamic output — these principles aren’t new. They are core design philosophies of web accessibility spanning decades.
Accessibility’s core premise is: you cannot assume through which channel users receive information. Some see with eyes, some listen with screen readers, some touch with braille displays. So content must be separated from presentation — content is semantic, presentation adapts dynamically to users’ abilities and devices.
The agent era faces the same structural problem, just in a different dimension. You cannot assume who is receiving information — it might be a human looking, an agent parsing, or another system consuming. And generative UI pushes one layer further: you cannot assume in what form information will be generated — the interface itself is dynamic, requiring multi-channel reachability on uncertain output.
WCAG’s POUR principles translate directly:
Perceivable: Skill output must be perceivable by both humans and agents simultaneously — semantic annotation solves this. Operable: Interactive component affordances must be reliable — interaction type declarations + unified client rendering. Understandable: Three-party cognitive alignment — Level 0 narrative shadow + snapshot mechanism. Robust: Cross-client compatibility — progressive enhancement’s layered architecture.
Level 0 narrative’s role is like alt text — not a degradation fallback, but a semantic anchor ensuring information remains reachable when any presentation form fails.
This domain’s design philosophy doesn’t need to be invented from scratch. The target audience extends from “people with different abilities” to the three-party relationship of “human + agent + system.” We can call this Agent Accessibility.
The Missing Specification
Synthesizing the above discussion, we can position generative UI within the agent ecosystem and identify a clear gap in the current specification landscape.
The capability layer is being defined — Anthropic’s skill spec and MCP define how skills are packaged and tools are called. The presentation layer is also in progress but fragmented — Google A2UI, Vercel JSON schema, and MCP Apps each have their own approach.
The middle layer — the interface between capability and presentation — nobody is defining.
This middle layer — the Skill-Surface Interface Spec — needs to cover five dimensions:
Semantic layer: Semantic annotation of skill output data, including confidence/uncertainty metadata, enabling the data model’s precision to be shared across all three parties.
Presentation layer: A skill’s positioning declaration on the spectrum — whether its output is suited for specialized tool rendering, parameterized component configuration, or needs a general canvas. Level 0/1/2 progressive enhancement, with Level 0 as a mandatory semantic anchor.
Interaction layer: Component interaction type declarations (display only / clickable / editable / destructive), affordance constraints, ensuring dynamically generated interfaces are predictable.
Communication layer: The dialog channel protocol between skill execution processes and dialog flows — message type definitions for emit (surface / progress / ask / result) and receive (user_input + snapshot / interaction / abort).
Alignment layer: Level 0 narrative shadow as a dual-channel verification mechanism, surface snapshots as a means of conveying user action state, and infrastructure for three-party cognitive synchronization.
Its design philosophy is rooted in the accessibility tradition: separating semantics from presentation, multi-channel reachability, progressive enhancement, and ensuring predictability on uncertain output. It doesn’t need to manage the extremes of the spectrum — specialized tools define everything themselves, and general canvas is too free to constrain. It manages the middle ground — the data contracts for parameterized components and the communication standards between skills and dialog flows.
This specification’s ecological niche is unique because those building the capability layer don’t think about presentation problems, and those building the presentation layer don’t think about skill execution context. The absence of this middle layer is the fundamental reason why skills and generative UI currently cannot be integrated.
The next article will start from specific technical frameworks — MCP Apps, Google A2UI, and others — analyzing their respective implementations, design trade-offs, and applicable boundaries, providing more concrete references for designing this Skill-Surface Interface.
2026.03