Tencent AI Open Sources Covo-Audio: A 7B Speech Language Mannequin and

Tencent AI Lab has launched Covo-Audio, a 7B-parameter end-to-end Giant Audio Language Mannequin (LALM). The mannequin is designed to unify speech processing and language intelligence by straight processing steady audio inputs and producing audio outputs inside a single structure.

System Structure

The Covo-Audio framework consists of 4 main elements designed for seamless cross-modal interplay:

Audio Encoder: The mannequin makes use of Whisper-large-v3 as its main encoder as a consequence of its robustness in opposition to background noise and various accents. This part operates at a body price of 50 Hz.
Audio Adapter: To bridge the encoder and the LLM, a specialised adapter employs three downsampling modules, integrating linear and convolution layers to cut back the body price from 50 Hz to six.25 Hz.
LLM Spine: The system is constructed upon Qwen2.5-7B-Base, which has been tailored to course of interleaved sequences of steady acoustic options and textual tokens.
Speech Tokenizer and Decoder: The tokenizer, based mostly on WavLM-large, makes use of a codebook measurement of 16,384 to supply discrete audio tokens at 25 Hz. The decoder employs a Circulation-Matching (FM) based mostly framework and a BigVGAN vocoder to reconstruct high-fidelity 24K waveforms.

A core contribution of this work is the Hierarchical Tri-modal Speech-Textual content Interleaving technique. Not like conventional strategies that function solely on the phrase or character degree, this framework aligns steady acoustic options $(a_c)$ , discrete speech tokens $(a_d)$ , and pure language textual content $(t)$ .

The mannequin makes use of two main patterns:

Sequential Interleaving $(a_c rightarrow t rightarrow a_d)$ : Steady options, textual content, and discrete tokens are organized in a progressive chain.
Parallel Integration $(a_c rightarrow t | a_d)$ : Steady options are aligned with a coupled text-discrete unit.

The hierarchical side ensures structural coherence by utilizing phrase-level interleaving for fine-grained alignment and sentence-level interleaving to protect international semantic integrity in long-form utterances^{. The coaching course of concerned a two-stage pre-training pipeline processing a complete of 2T tokens^.}

Intelligence-Speaker Decoupling

To mitigate the excessive price of establishing large-scale dialogue information for particular audio system, the analysis crew proposed an Intelligence Speaker Decoupling technique. This method separates dialogue intelligence from voice rendering, permitting for versatile voice customization utilizing minimal text-to-speech (TTS) information.

The tactic reformats high-quality TTS recordings into pseudo-conversations with masked textual content loss^{. By excluding the textual content response portion from the loss calculation, the mannequin preserves its reasoning talents whereas inheriting the naturalness of the TTS speaker^{. This permits customized interplay with out the necessity for intensive, speaker-specific dialogue datasets^.}}

Full-Duplex Voice Interplay

Covo-Audio advanced into Covo-Audio-Chat-FD, a variant able to simultaneous dual-stream communication^{^{^{^{. The audio encoder is reformatted right into a chunk-streaming method, and the person and mannequin streams are chunk-interleaved in a 1:4 ratio^{. Every chunk represents 0.16s of audio^.}}}}}

The system manages conversational states by way of particular architectural tokens:

THINK Token: Signifies a listening-only state whereas the mannequin waits to reply.
SHIFT Token: Signifies the transition to the mannequin’s talking flip.
BREAK Token: Detects interruption alerts (barge-ins), triggering the mannequin to terminate talking instantly and swap again to listening.

For multi-turn situations, the mannequin implements a recursive context-filling technique, the place steady audio options from person enter and generated tokens from earlier turns are prefixed as historic context^.

Audio Reasoning and Reinforcement Studying

To boost complicated reasoning, the mannequin incorporates Chain-of-Thought (CoT) reasoning and Group Relative Coverage Optimization (GRPO). The mannequin is optimized utilizing a verifiable composite reward operate:

$$R_{complete} = R_{accuracy} + R_{format} + R_{consistency} + R_{considering}$$

This construction permits the mannequin to optimize for correctness $(R_{accuracy})$ , structured output adherence $(R_{format})$ , logical coherence $(R_{consistency})$ , and reasoning depth $(R_{considering})$ .

Analysis and Efficiency

Covo-Audio (7B) reveals aggressive or superior outcomes on a number of evaluated benchmarks, with strongest claims made for fashions of comparable scale and chosen speech/audio duties. ^{^{^{^{On the MMAU benchmark, it achieved a median rating of 75.30%, the best amongst evaluated 7B-scale fashions^{^{^{^{. It notably excelled in music understanding with a rating of 76.05%^{. On the MMSU benchmark, Covo-Audio achieved a number one 66.64% common accuracy^.}}}}}}}}}

Concerning its conversational variants, Covo-Audio-Chat demonstrated robust efficiency on URO-Bench, significantly in speech reasoning and spoken dialogue duties, outperforming fashions like Qwen3-Omni on the Chinese language observe^{. For empathetic interplay on the VStyle benchmark, it achieved state-of-the-art leads to Mandarin for anger (4.89), unhappiness (4.93), and anxiousness (5.00)^.}

The analysis crew notes an ‘early-response’ difficulty on the GaokaoEval full-duplex setting, the place unusually lengthy silent pauses between vocal fragments may cause untimely responses. This ‘early-response’ habits correlates with the mannequin’s pause-handling success metric and is recognized as a vital course for future optimization.

Key Takeaways

Unified Finish-to-Finish Structure: Covo-Audio is a 7B-parameter mannequin that natively processes steady audio inputs and generates high-fidelity audio outputs inside a single, unified structure. It eliminates the necessity for cascaded ASR-LLM-TTS pipelines, lowering error propagation and knowledge loss.
Hierarchical Tri-modal Interleaving: The mannequin employs a specialised technique to align steady acoustic options, discrete speech tokens, and pure language textual content. By interleaving these modalities at each phrase and sentence ranges, it preserves international semantic integrity whereas capturing fine-grained prosodic nuances.
Intelligence-Speaker Decoupling: Tencent analysis crew introduces a method to decouple dialogue intelligence from particular voice rendering. This enables for versatile voice customization utilizing light-weight Textual content-to-Speech (TTS) information, considerably decreasing the price of growing customized conversational brokers.
Native Full-Duplex Interplay: The Covo-Audio-Chat-FD variant helps simultaneous listening and talking. It makes use of particular architectural tokens—THINK, SHIFT, and BREAK—to handle complicated real-time dynamics corresponding to easy turn-taking, backchanneling, and person barge-ins.
Superior Parameter Effectivity: Regardless of its compact 7B scale, Covo-Audio achieves state-of-the-art or extremely aggressive efficiency throughout core benchmarks, together with MMAU, MMSU, and URO-Bench. It incessantly matches or exceeds the efficiency of a lot bigger methods, corresponding to 32B-parameter fashions, in audio and speech understanding duties.

Try the Paper, Model on HF and Repo. Additionally, be at liberty to comply with us on Twitter and don’t overlook to affix our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Michal Sutter is an information science skilled with a Grasp of Science in Information Science from the College of Padova. With a stable basis in statistical evaluation, machine studying, and information engineering, Michal excels at reworking complicated datasets into actionable insights.

Source link