Gander: Unified End-to-End AI Agent for Perception, Interaction, and Agency

A new agent built for continuous collaboration

Gander, newly presented in the “Omni Interaction Agent Technical Report,” is an end-to-end AI agent designed to keep perceiving, conversing, and acting within the same continuous framework rather than switching between isolated chat, vision, speech, and tool-use modules [1]. The authors describe the system as a response to a core limitation of many AI assistants: they may be powerful planners or fluent talkers, but they usually operate in discrete request-and-response turns rather than in the fluid style of human collaboration [1].

The release matters because Gander is explicitly framed not as a voice wrapper around an existing agent, but as an “omni interaction agent” that continuously receives speech, video, and text while supporting full-duplex dialogue and asynchronous task execution [1]. In practical terms, the user can speak, interrupt, share a screen or camera feed, ask for progress, add constraints, or cancel a task while the system continues tracking the larger objective [2]. That makes the project a notable step toward AI assistants that do not simply answer prompts, but remain present during an evolving activity.

The working headline for this article is the story itself: Gander is a unified end-to-end AI agent for perception, interaction, and agency. The technical report was posted on September 8, 2026, and the official project blog lists a September 9, 2026 report date, placing the release squarely inside the current 72-hour news window [1][2].

What Gander unifies

The key claim is unification. Gander combines omni perception, real-time interaction, and agentic capabilities in one architecture [1]. “Omni” here means that the system can use speech, video, and text as continuing streams rather than as one-off attachments [1]. The official project page describes the same idea in user-facing terms: Gander follows spoken requirements, the visual scene in front of the user, and later feedback as parts of one task process [2].

This is a different design target from a conventional chatbot. A standard assistant waits for a full user turn, generates a complete answer, and then waits again. Gander instead models interaction as a timeline. It can decide whether to listen, speak, interrupt, or delegate work to a background agent while new sensory input continues arriving [1][3]. That is why the release repeatedly emphasizes “full-duplex” interaction: the model is expected to listen while speaking and to handle backchannels, interruptions, and overlapping signals rather than treating them as errors [1].

The demos described on the official project page show the intended interaction pattern. A user can ask Gander to create an HTML Tetris game, add a restart function, and then use a shared screen to request a color change for the pause button; the spoken instructions and visual feedback remain tied to the same task [2]. Another demo has a user share a technical report and ask Gander to conduct research and summarize it, while the system combines document context with spoken guidance to launch a background task [2]. These examples illustrate the central concept: the conversation is not separate from the work; it is the control surface for the work.

The “Cerebellum-Brain” split

Gander’s architecture rests on a division between a front “Cerebellum,” an agent orchestration runtime, and a back “Brain” [1][3]. The front Cerebellum handles real-time audio-visual perception, interaction control, text generation, and speech synthesis [3]. The back Brain is responsible for long-horizon reasoning, tool use, file and application operations, and workflow execution [3]. Between them, the orchestration runtime manages trusted turns, task state, multimodal context, worker events, permissions, and delivery [3].

The analogy is useful because it separates reaction from deliberation. Human conversation requires quick timing: knowing when to pause, when to acknowledge, when to stop speaking, and when to ask for clarification. Long-running tasks require a different rhythm: planning, searching, editing files, using tools, and reporting progress. Gander’s design keeps those processes connected while allowing each to specialize [1][3].

The GitHub release describes a live request as a causal path: browser microphone audio and optional camera or screen frames stream into the system; the Cerebellum predicts whether to listen, speak, interrupt, or invoke a task operation; the runtime binds agentic work to the finalized user turn; and the Brain works asynchronously while milestones, questions, permissions, and results return through the same live conversation [3]. That workflow is the clearest expression of Gander’s agency claim. The agent is not only deciding what to say; it is deciding whether to act, how to bind an action to a user’s intent, and how to keep the user involved while execution continues.

The back Brain is also designed to be pluggable. The released repository names Codex as the included Brain provider, but says provider construction is registry-based so another agent can be integrated without changing the Cerebellum task protocol or gateway state machine [3]. This makes Gander less a single monolithic assistant than a pattern for connecting a real-time multimodal interface to interchangeable reasoning and tool-use backends.

The Thinker-Talker streaming model

Inside the Cerebellum, Gander uses a streaming Thinker-Talker architecture [1]. The Thinker controls interaction and content, while the Talker renders speech without blocking perception [3]. The technical report says incoming observations, user inputs, interaction decisions, and model outputs are flattened into an ordered token stream at chunk level, giving the system a shared representation for low-latency continuous interaction [1].

The repository explains the released temporal contract more concretely: Gander flattens continuous interaction into one-second causal units, with newly available audio and visual observations preceding the model decision [3]. In each unit, the Thinker predicts a control behavior such as listen, speak, interrupt, or tool, and then produces text or structured calls if needed [3]. When the system speaks, the Talker conditions on Thinker hidden states and aligned text tokens to generate speech tokens, while a causal flow-matching decoder renders audio incrementally [3].

The released alignment is eight text tokens to 50 S3 speech tokens per speaking unit, and the Talker can be detached so perception continues while speech is generated [3]. The project page also notes a one-second streaming unit and a 128-chunk sliding context, while cautioning that these are model settings rather than an end-to-end latency measurement [2]. For readers evaluating deployment claims, that caveat matters: Gander is designed for streaming behavior, but the project is not presenting the one-second unit as a direct user-perceived latency number.

Training data and evaluation signals

Gander starts from MiniCPM-o 4.5 and is trained through a two-stage process focused first on the Thinker and then on the Talker [2][3]. The official project page says the interaction corpus contains approximately 2.7 million examples covering speech, audio-visual and agentic interaction, robustness, and negative supervision [2]. The repository breaks that mixture into speech interaction, audio-visual interaction, agentic interaction, and robustness or negative examples [3].

That training design reflects the problem Gander is trying to solve. It is not enough for a system to answer well when asked a clean question. It must also learn when not to answer, how to ignore irrelevant voices, how to handle backchannels, how to ask for missing information, and how to keep task progress synchronized with conversation [2][3]. The official project demos include irrelevant speech during a passport-loss conversation, multi-speaker travel planning, simultaneous interpretation, proactive visual alerts, and spoken commonsense reasoning [2].

On evaluation, the technical report says Gander is tested across conversational ability, omni understanding, interactive capability, and agentic intelligence [1]. The repository summarizes several headline results: across 2,052 benchmark utterances, Gander leads the full-duplex streaming group on both SpokenQA subsets and places second on both VoiceBench subsets; it reports 78.53% accuracy on Daily-Omni after full-duplex and agentic post-training; and on Full-Duplex-Bench v3 it reports 100% appropriate turn-taking with 8.0% premature interruption [3]. It also reports that delegated Full-Duplex-Bench scenarios produced 45 out of 45 final responses correctly bound to the relevant task [3].

These numbers should be read as research-release signals, not as proof of production reliability. The report itself says the field lacks a dedicated benchmark for omni interaction agents and that existing benchmarks evaluate omni understanding, duplex interaction, and agentic execution mostly in isolation [1]. That limitation is important because Gander’s central promise lies precisely in the interaction among those capabilities.

What is available now

The project is being released as an open research system. The technical report says the team is releasing Gander with models, code, and data to facilitate community research and development [1]. The GitHub repository states that Gander was officially released on September 9, 2026, and adds on September 10, 2026 that the dataset is undergoing the open-source release process and will be made publicly available soon [3]. The repository is under an Apache-2.0 code license, while the quick-start instructions describe a full browser system with managed ASR, a streaming Thinker, a detached Talker, the agent runtime, a web UI, and a Codex-backed Brain [3].

The practical requirements are nontrivial. The release profile calls for Linux, Conda, an NVIDIA driver compatible with the CUDA 12.4 PyTorch stack, and three physical GPUs: one each for the Thinker, Talker, and ASR [3]. It also requires model downloads for MiniCPM-o 4.5, Gander, and faster-whisper large-v3, plus an authenticated Codex CLI or compatible executable for the Brain [3]. That makes the first audience researchers and advanced developers, not casual users.

Why it matters

Gander’s significance lies in its architecture rather than in a single benchmark score. The project argues that interactivity should be a native model capability, not only an external voice-activity detector, ASR transcript, or orchestration layer wrapped around a planner [1]. By binding continuous perception, turn timing, speech generation, and task delegation to one timeline, it points toward agents that can remain responsive while acting autonomously.

The hard problems remain. The report identifies future challenges in data and model scaling, post-training and reinforcement learning for long-horizon omni scenarios, richer bidirectional communication between Brain and Cerebellum, memory, and unified evaluation [1]. Those are not minor footnotes; they define whether systems like Gander can move from compelling demos to reliable assistants that safely act across real user environments.

For now, Gander is best understood as a fresh research blueprint for AI autonomy. It combines a live perceptual interface, learned real-time conversational control, and asynchronous agency in one system. If the broader field adopts similar designs, the next generation of agents may feel less like tools waiting for prompts and more like collaborators that can see, hear, speak, and work at the same time.

Sources from the last 72 hours

1. [1]Omni Interaction Agent Technical ReportSep 8, 2026, 4:22 PM UTC
2. [2]GitHub - Omni-Interaction-Gander/Omni-Interaction-Agent: Open-source end-to-end omni interaction agent for natural full-duplex voice collaboration, continuous audio-visual perception, and asynchronous long-horizon agentic execution.Sep 9, 2026, 12:00 AM UTC
3. [3]Gander：全模态交互智能体 · 技术博客Sep 9, 2026, 12:00 AM UTC

AI-generated article based on recent web research, then preserved as a dated editorial snapshot.