On this page
Several months ago, I thought the central problem of full-duplex voice modeling was timing. Audio would stream continuously into a model, and the model would learn when to call a single function: respond to the user.
This post is also available in Chinese (中文版).
That framing now feels incomplete.
A useful voice agent does not merely decide when to speak. It listens while actions are running, receives results from tasks it started earlier, handles interruptions and corrections, and decides when to speak, act, delegate, wait, or ask for clarification. Its inputs are not only audio tokens, but events from an external world.
After spending the past few months working on these systems, I have become less convinced that the first-order problem is the model architecture or the optimization algorithm. The harder question is what environment is worth optimizing in the first place. Without a concrete product, it is difficult to define the right behavior, build the right harness, or produce the data the model needs to learn.
A Co-worker in a Shared Workspace
Imagine that I am writing a design document with a voice agent beside me. We share the same workspace: it can see the paragraph I am reading, the section I selected, the edits I just made, and the tasks already running in the background.
I say:
This introduction still feels too abstract. I do not just mean that the model can listen and speak at the same time. I mean that I should still be able to change its direction while it is working.
This is not a clean command. I may be thinking out loud, asking for a rewrite, or inviting a discussion. The agent has to infer my implied intent without treating every thought as permission to act.
It might reply briefly:
Right—the important property is steerability during execution. I can rewrite this paragraph around that distinction.
If I agree, it can update the document. At the same time, it might delegate a longer task to collect examples or develop a diagram. If I then interrupt and say, “Do not frame it around interruption alone; delegation is equally important,” the agent should update the document, steer the task already in progress, and make sure the old result does not later overwrite the new direction.
This is the product shape I have in mind: not a voice interface attached to a chatbot, but a voice-native co-worker operating over a shared workspace. The workspace could be a document, a codebase, a canvas, or another environment where ideas become persistent and actionable.
Audio Is an Infinite Stream
Full duplex starts with an always-on assumption. During an active session, audio keeps arriving. The stream may contain speech, silence, hesitation, background noise, another speaker, or an interruption. It does not wait for a clean turn boundary.
The model therefore has to perceive and act over an effectively unbounded timeline. Even without tools, this is difficult. It has to decide whether the user has finished speaking, distinguish an interruption from a backchannel, ignore irrelevant noise, track speaker identity, and produce a response of the right length at the right moment.
Current real-time systems already demonstrate much of this behavior, but making it consistently reliable is still hard. The last mile includes conversational rhythm, personalization, and alignment with individual preferences: when to be concise, when to elaborate, when to be proactive, and when to stay quiet.
Those problems matter, but they are not my main focus here. The more useful abstraction is that the model continuously receives observations and chooses actions. Speech is one observation channel, and speaking is one possible action. Once the agent is connected to a workspace, they are no longer the only ones.
From One Function to an Action Space
Before working on voice agents, I implicitly modeled the assistant as having one function:
respond_to_user()
Full-duplex modeling made this function more flexible. It could be triggered after a complete request, during a hesitation, or when the model believed a small proactive response would help. But the assistant was still fundamentally a conversational system.
A workspace changes the action space:
speak()
inspect_workspace()
edit_document()
call_tool()
delegate_task()
steer_task()
cancel_task()
wait()
The user remains the most important part of the environment, but not the only part. The agent also interacts with documents, code, tools, APIs, and delegated workers. Each interaction produces new events that may arrive while the user is still speaking or while the agent is doing something else.
Even speak() is better understood as an ongoing activity than a completed response. Speech can begin before the full answer is generated. It can be interrupted, shortened, or redirected. It changes what the user knows about the system and creates an opening for correction.
This is where voice and agency begin to converge. The agent is not alternating between conversation and work. It is communicating while work is in progress and remaining steerable throughout.
Three Concurrent Loops
In a workspace co-worker, I find it useful to think about three loops operating at different time scales.
The conversation loop handles listening, short acknowledgements, clarification, turn-taking, and interruption. It needs to feel immediate.
The workspace loop reads the current artifact and makes local changes: opening a file, inspecting a selection, editing a paragraph, or showing a diff. These operations should usually return quickly enough to remain part of the live interaction.
The delegation loop handles work that may take seconds or minutes: researching a question, running code, producing a larger implementation, or coordinating with another system. These activities need identity and state. They may be running, blocked, completed, failed, steered, or cancelled.
The boundary is not simply latency. A fast action may have an irreversible side effect, while a slow task may be easy to preview and undo. What matters is the contract around the action: how long it takes, whether it can be interrupted, whether it can be steered, what intermediate progress is visible, and what happens if the user’s intent changes.
A robust harness has to coordinate all three loops. While a delegated task runs, the agent should continue listening. While the agent speaks, a tool result may arrive. While a document edit is being prepared, the user may manually change the same paragraph. The system must reconcile these events without losing the conversational thread or corrupting the workspace.
The Voice Agent Is an Event Loop
This leads to the mental model I now prefer: a voice agent is an always-on event loop.
Audio provides the main timeline, but other events can enter at any point:
audio chunk
user interruption
workspace update
tool result
delegated-task progress
task completion
task failure
The agent can also initiate several activities before earlier ones have finished. Their results may return out of order. A result may correspond to a request from much earlier in the conversation. It may also be stale because the user has already changed direction.
The challenge is therefore not only function calling. It is maintaining causal and conversational consistency across asynchronous event streams.
The model needs to understand what it requested, which task an event belongs to, whether that task is still relevant, and how the new information should affect both the workspace and the conversation. Sometimes it should surface a result immediately. Sometimes a one-sentence progress update is enough. Sometimes it should quietly update its state and wait for a better moment.
This is also why implied intent matters. Inferring that the user probably wants something is not the same as receiving permission to do it. The policy should depend on confidence, cost, reversibility, and user preference. A low-risk draft can be produced proactively. Sending a message or making an irreversible change should require a much clearer commitment.
The right amount of initiative cannot be defined in the abstract. It depends on the product and the people using it.
The Harness Produces the Data
This has been the largest change in my thinking over the past few months.
There are many plausible technical implementations for real-time interaction, tool use, and delegation. Given enough representative data, several of them may reach a reasonable baseline. The more difficult problem is creating that data.
Useful training examples are not just pairs of user speech and assistant responses. They include workspace state, active tasks, partial results, interruptions, corrections, delays, failures, and actions with real consequences. The model has to experience cases where a user changes direction mid-sentence, a cancelled task still returns, two delegates produce conflicting results, or an apparently helpful intervention turns out to be unwanted.
Synthetic data can cover some of this, but the event distribution ultimately comes from the product. A document co-worker, a coding agent, and a general desktop assistant expose different state, permit different actions, and require different forms of initiative. They need different harnesses and therefore generate different learning problems.
The relationship is circular. A product scenario shapes the harness, which produces interactions and failures, which become training data and model behavior. User feedback then closes two loops: it reveals how the product and harness should change, and it identifies behaviors that should be reinforced, corrected, or reevaluated in the model.
The harness is not only where the model is deployed. It is also the data generator.
Building a realistic harness often looks like unglamorous engineering: tracking task state, handling race conditions, defining cancellation semantics, protecting user edits, and recovering from partial failure. But without this work, the model is trained on an artificial world and evaluated against behaviors that may not matter in practice.
This is why finding a concrete scenario may be more important than choosing a training algorithm. The product defines what good behavior means.
From Executor to Co-worker
Reliable orchestration is still only the foundation.
The co-worker I ultimately want is not just an executor that accepts commands and returns artifacts. It should help me think. That difference becomes clear in longer sessions.
Suppose we have spent an hour developing a design. I briefly considered one approach, rejected it because of a latency constraint, and moved on. Later, a delegated task returns with evidence that changes that constraint. An executor reports the result. A co-worker understands why the result matters: it reconnects the abandoned approach to the decision we are making now. It might say, “This changes the latency assumption that made us reject the streaming design earlier. Do you want to revisit it?”
Or suppose several tasks are running while I focus on writing. I do not want a notification for every intermediate result. I want the agent to preserve my concentration, combine related updates, and interrupt only when a result blocks the current plan or creates a meaningful choice. The right behavior depends on what I am doing, how urgent the result is, and how I prefer to work.
These examples require more than a long context window. The agent needs a structured understanding of the workspace: which decisions are active, why they were made, which tasks depend on them, and what the user is paying attention to now. It also needs judgment about initiative. Remembering everything is not useful if the agent surfaces the wrong memory at the wrong time.
That introduces a second layer of difficult problems: context management, memory, personalization, and human preference alignment. A useful co-worker needs to understand not only the current request, but the user’s goals, working style, and desired level of initiative. The right behavior in a coding session may be wrong in a planning discussion. The right behavior for one person may feel intrusive to another.
I do not think these questions can be answered before the product exists. “Be proactive” is not a useful target by itself. In a shared document, proactivity might mean quietly preparing a reversible draft. In a coding workspace, it might mean running a relevant test after an edit. Sending a message, changing a deployment, or redirecting a long-running task requires a different boundary.
We need to observe where users welcome help, where they want control, and where proactive behavior becomes distracting. Personalization is not a feature that can be added independently of the interaction design; it emerges from a concrete relationship between a user, an agent, and a shared environment.
The Model Was Not the First Question
I started this work by asking how to build a better full-duplex model. I now think that question comes too early.
Early on, it was natural to focus on the most legible technical choices: how audio and text should be interleaved, when the model should emit a response, which optimization method to use, and how to reduce latency. Progress on those questions was real, but it did not automatically produce a convincing co-worker.
Some of the most visible failures happened one layer above the model. A task completed correctly but returned after the user had changed direction. The agent produced a good answer but spoke for too long while the user was trying to interrupt. A delegate made progress, but the main agent did not know whether the update was important enough to surface. A workspace edit was locally reasonable but inconsistent with a decision made twenty minutes earlier.
It is tempting to label all of these as model failures. Sometimes they are. But often the system has never been given a precise task contract, a reliable representation of workspace state, or training examples that show the desired behavior. Improving the model without improving that environment optimizes around an undefined target.
The first question is therefore what kind of collaboration we want to make possible. For a workspace co-worker, success is not only answer quality. It is whether the user can stay in flow, redirect work cheaply, understand what is happening, and trust that the shared artifact still reflects their intent. These product-level outcomes tell us which actions to expose, which events to preserve, what the harness must guarantee, and what the model should learn.
This is a loop rather than a fixed sequence. User feedback may reveal a behavior that needs better training, but it may also reveal that the product scenario was wrong: the agent had too much authority, the workspace exposed too little context, or the interaction demanded a confirmation at the wrong moment. The next model and the next product should be designed together.
The model still matters. Conversational quality, low latency, long-context understanding, and stable asynchronous reasoning are all difficult technical problems. But they become much easier to reason about once they are attached to a real environment and a real user need.
For me, that has been the main lesson of the past few months: the path to a better voice agent may begin not with a new model, but with a workspace worth sharing.