On August 5th, ByteDance Seed released the native audio-video full-duplex (a two-way channel that allows simultaneous listening and speaking, unlike walkie-talkies which require taking turns) large model SeedRealtime: a unified architecture that simultaneously integrates audio, video, and text, enabling real-time interaction across continuous streams of visuals and sounds—officially described as "watching, listening, and speaking simultaneously." This is not just a paper launch: it has been fully deployed in the Doubao App, where users can experience it by selecting "phone call" in the chat box to enter a video call.
Three Core Breakthroughs, Already Fully Deployed in Doubao
The official release outlines SeedRealtime's capabilities in three core breakthroughs, each addressing the "real-time interaction" challenge.
First, joint audio-video understanding. Sound, visuals, and temporal information are deeply integrated: when encountering homophones, it can resolve ambiguity by combining the current visual context; if you say "how do I do this," it can determine what "this" refers to by following your gestures, gaze, and the scene.
Second, proactive interaction. It no longer waits passively for your questions: when a key target appears on screen (for example, if you instruct "remind me when you see that exhibit"), it will speak up; it can also call upon tools and incorporate the retrieved information into its response.
Third, rhythm control. It can sense your conversation state in real-time, naturally interjecting when appropriate and staying silent when not to interrupt; more crucially, it has anti-interference capabilities—able to distinguish between background noise and unrelated chatter, avoiding being triggered by irrelevant sounds.
Why is "Watching, Listening, and Speaking Simultaneously" So Difficult?
To understand the significance of this release, let's first look at the two previous approaches and where they stumbled.
First approach: Cascade systems(breaking tasks into a series of modules that pass the work along: first listen, then see, finally speak). Speech is first handed over to ASR(Automatic Speech Recognition, converting speech to text) to be converted into text, then passed to a visual language model for understanding, and finally synthesized into speech by TTS(Text-To-Speech, converting text back to speech)—modules are connected in series, delays are accumulated, and information is lost at each stage.
Second approach: End-to-end models(a model that goes directly from input to output without modular relay). More fluent, but the official release points out that many solutions still rely on external VAD(Voice Activity Detection, using external rules to determine when you have finished speaking) to judge turns—essentially still a half-duplex(a channel where only one party can speak at a time) question-and-answer format.
SeedRealtime's solution is to unify sound, visuals, timing, and expression into a single end-to-end model: rather than listening first, then watching, and finally responding, it perceives, understands, decides, and expresses simultaneously.
The Challenge Isn't Hearing Clearly, It's Knowing When to Speak
The most counterintuitive point in the official technical breakdown is: speech is actually easier, video is the real challenge.
Speech naturally has pauses, allowing the model to use these pauses to judge "you're done, now it's my turn." But video is always on, continuously changing: the model must constantly understand what is happening in the scene, without being triggered by every minor disturbance—it must continuously judge: which object to focus on, who to listen to, whether it should respond at this moment.
This is the value of "anti-interference." The official examples include two very illustrative cases: at Daxing Airport, when a companion casually mentions "Laoli's flight," the model is not triggered by this unrelated conversation; only when the user formally asks does it combine the information from the big screen it previously saw to respond. When accompanying a child learning English, with the father in the background on a call and constant human voices, the model consistently follows the child's finger to correct pronunciation and construct sentences—it doesn't hesitate when it's time to speak, nor does it get distracted when interfered with.
The technical threshold of full-duplex is not "how clearly it hears," but "when to speak." Bringing turn judgment from external rules into the model itself is the real hurdle that this release has overcome.
Three of the Seven Official Examples Worth Noting
The release provides seven real-world scenario examples. Aside from the straightforward capabilities like "recognizing people" and "translating," three examples best illustrate the value of "watching, listening, and speaking simultaneously":
Museum "Target Reminder": You instruct "remind me when you see the错金银铜虎噬鹿屏座," it continuously monitors the scene as the camera moves, and upon spotting the exhibit, it speaks up and then explains the artifact's craftsmanship—the task is preserved in the context, and the target triggers the response.
Coffee Machine "Real-time Correction": You pour whole coffee beans directly into the extraction handle, it immediately points out "you need to grind them into fine powder first"; after extraction, it observes the cup's oil content and proactively suggests "shorten the time by 2-3 seconds next time"—no one asked, it proactively comments based on the visual input.
Airport "Chit-chat Memory": A companion casually mentions "Laoli's flight," it doesn't get triggered, but it remembers; when you formally ask, even though the flight information has moved off the screen, it still retrieves the big screen information to answer and supplements it with the location of the luggage carousel from the internet.
These three cases share a commonality: the model is no longer "only answers when asked," but rather continuously observes and intervenes at the right moment. The two official demo videos below correspond to the museum reminder and the airport small-talk memory. Of course, all are official demonstration scenarios, and the real-world failure rate awaits your own testing.
Official demo ①: museum “target reminder” — speaks up and explains the exhibit the moment the camera sweeps past it · Source: ByteDance Seed official release
Official demo ②: airport “small-talk memory” — ignores unrelated chatter, then recalls flight info seen earlier when formally asked · Source: ByteDance Seed official release
You Can Try It Today, But Test It with "Noise"
The barrier to entry is low, but "how to test" requires some consideration.
Entrance: Update the Doubao App to the latest version, select "phone call" in the chat box, and enter the video call interface.
Test Anti-interference: Find a real noisy environment (dinner party, subway, babysitting), turn on background noise and chat with it, see if it gets distracted by nearby chatter or interrupts randomly.
Test Proactivity: Give it a task like "remind me when you see XX," see if it can accurately trigger when the scene changes.
Remember the Metric: "Conversation rhythm issues reduced by half" is the official end-to-end human evaluation claim, with no third-party verification yet; your actual experience is more reliable than any benchmark.
The hurdle SeedRealtime has overcome is not "how clearly it hears," but "when to speak." Bringing vision, hearing, and speaking into a single end-to-end model, allowing turn judgment to shift from external rules to the model's own continuous decision-making—this is the watershed moment from "walkie-talkie AI" to "face-to-face AI." Doubao has fully deployed it, just select "phone call" in the chat box to try; it is recommended to test it with your own noisy daily life, don't trust the demos.
The facts in this article are from the ByteDance Seed official release (2026-08-05). The claim of "conversation rhythm issues reduced by half" is based on the official end-to-end human evaluation metric, with no third-party independent verification; the seven cases are all official demonstration scenarios. Experience entrance: Doubao App latest version "phone call" video call.