One-sentence Summary

MOSI.AI presents MOSS Transcribe Diarize, an end-to-end unified multimodal large language model for Speaker-Attributed, Time-Stamped Transcription that supports up to 90-minute audio via a 128k context window, delivers strong long-range speaker memory and direct timestamp generation, and outperforms top commercial systems on multiple public and in-house benchmarks.

Key Contributions

• MOSS Transcribe Diarize is a unified multimodal model that jointly performs word recognition, speaker attribution, and timestamp prediction in a single end-to-end pass, eliminating separate subsystems.
• The model employs a 128k-token context window to process up to 90-minute meetings without chunking, preserving long-range speaker consistency and avoiding boundary artifacts.
• Training on extensive in-the-wild conversations and property-aware simulated mixtures enables robust generalization, and the model outperforms state-of-the-art commercial systems on AISHELL-4, Podcast, and Movies in CER, cpCER, and Δcp.

Introduction

Accurate speaker-attributed, timestamped transcripts of multi-speaker conversations are essential for meeting assistants, call-center analytics, and assistive technologies, yet current solutions typically stitch together separate ASR and diarization modules, which suffer from cascading errors and inconsistent global context. Although recent multimodal LLMs move toward joint modeling, they are still constrained by short context windows, fragile long-range speaker memory, and an inability to natively emit segment-level timestamps, forcing chunk-wise processing and external alignment that reintroduce modular mismatches. The authors introduce MOSS Transcribe Diarize, a unified multimodal large language model that performs word recognition, speaker attribution, and timestamp prediction in a single end-to-end pass, leveraging a 128k-token context window to process up to 90 minutes of audio without chunking and thereby preserving discourse continuity and speaker consistency.

Dataset

The authors build a multi-speaker dataset that combines real-world recordings and simulated mixtures, then evaluate on three diverse benchmarks.

• Real data sources

  • AISHELL-4: a public meeting-room corpus with far-field overlapping audio and near-field recordings. The authors average the far-field channels and use the same channel for both training and evaluation.
  • Podcast test set: curated from high-quality, multi-guest YouTube interviews. The platform’s subtitles serve as reference transcripts.
  • Movies test set: short segments from online films and TV series, rich in overlapping speech, primarily in Chinese and English but also covering Korean, Japanese, and Cantonese. All samples are manually annotated. The Podcast and Movies sets will be open-sourced on Hugging Face.

• Simulated data (training only)

  • Generated from an in‑house pool of single‑speaker utterances.
  • A probabilistic simulator creates each synthetic dialogue by selecting 2–12 distinct speakers, one utterance per speaker, and partitioning each utterance into contiguous word runs (sampled segment count and log‑normal weights).
  • Segments are placed on a single timeline with Gaussian‑distributed inter‑segment gaps, enforcing speaker alternation and allowing overlaps up to 80% of the shorter segment.
  • Segments boundaries are snapped to low‑energy points and a 50 ms cross‑fade is applied.
  • The mixtures are augmented with real‑world noise and reverberation (SNR uniformly sampled from 0–15 dB).

• How the data is used

  • AISHELL-4 is used for both training and evaluation (far‑field averaged channel).
  • Simulated mixtures are added to training to strengthen speaker attribution and timestamp prediction.
  • The Podcast and Movies sets are held out exclusively for evaluation.

Method

The authors propose MOSS Transcribe Diarize, an end-to-end model designed for unified long-context modeling. As shown in the figure below:

The architecture couples an audio encoder with a projection module that maps multi-speaker acoustic embeddings into the feature space of a pretrained text large language model (LLM). This design enables the backbone to jointly align speaker identities with lexical content. To handle temporal information effectively over long durations, the authors represent time explicitly as formatted timestamp text inserted between audio encoder chunks. This textual token-based time encoding avoids binding temporal encoding to absolute positional indices, which can become sparse and ineffective over long durations, thereby enabling accurate timestamp generation and stable speaker attribution over hour-scale audio.

To strengthen speaker attribution and timestamp prediction during training, and to address the scarcity of high-quality real-world recordings, the authors utilize simulated data. They construct synthetic multi-speaker conversational data using a controllable probabilistic simulator. For each synthetic dialogue, 2 to 12 distinct speakers are drawn, and one utterance per speaker is randomly selected from an in-house corpus. Each utterance is partitioned into contiguous word runs by sampling a segment count and log-normal weights. These segments are placed on a single timeline with Gaussian-distributed inter-segment gaps, enforcing speaker alternation while permitting overlaps capped at 80 percent of the shorter segment. To improve perceptual continuity, segment boundaries are snapped to nearby low-energy points and 50 ms cross-fades are applied. Finally, the mixtures are augmented with real-world noise and reverberation, with signal-to-noise ratios sampled uniformly from 0 to 15 dB.

Experiment

The evaluation covers three diverse benchmarks representing long-form meetings, multi-speaker podcasts, and short overlap-rich movie segments, using CER, cpCER, and their difference Δcp to assess both transcription and speaker attribution. MOSS Transcribe Diarize outperforms all closed-source baselines, especially on long-form audio where some models fail due to input length constraints, and it consistently yields minimal speaker-attributed degradation. The results confirm the effectiveness of end-to-end long-context modeling for robust joint speaker-attributed recognition.

The evaluation employs three datasets with contrasting characteristics. AISHELL-4 Test offers long real-world conference recordings averaging 38 minutes and 5 to 7 speakers. The Podcast dataset contains even longer, high-quality multi-guest interviews with up to 11 speakers, while the Movies dataset comprises short, overlapping multilingual segments averaging 11.5 seconds. AISHELL-4 Test features the longest continuous recordings, with an average duration of 2290.6 seconds (nearly 40 minutes) and 5 to 7 speakers. Podcast includes the widest speaker count range (2 to 11) and the longest individual samples, up to 3636.5 seconds. Movies consists of very short clips (average 11.5 seconds) spanning 1 to 6 speakers, designed for dense overlap and multilingual scenarios. The Podcast and Movies datasets are internally curated and will be open-sourced on Hugging Face.

The evaluation setup uses three contrasting datasets to test system performance across varied real-world conditions. AISHELL-4 Test offers long conference recordings averaging nearly 40 minutes with 5 to 7 speakers, Podcast provides even longer multi-guest interviews with up to 11 speakers, and Movies consists of very short, overlapping multilingual segments. This configuration validates robustness to extended recording durations, wide speaker counts, and dense speech overlap.