SGLang Omni TTS Runtime

Make CosyVoice3 speech generation faster by improving SGLang-Omni's serving runtime, while protecting speech quality and checking Qwen Base and MOSS for regressions.

The climb

The dots show actual CosyVoice3 throughput from 11 complete Judge measurements, including evaluations that received no reward. Hollow dots did not pass all original gates; the filled dot did. Each colored line shows the best measured throughput within one research segment, not accepted reward. The three protocols differ, so their lines are not joined. The dashed reference applies only to the later segments; the initial segment has no matching baseline recorded in its feedback. Time is measured from the first segment and includes the gaps between segments.

The task

Make the same speech models serve requests faster, without trading away correctness or speech quality. The primary objective is CosyVoice3 throughput; Qwen3-TTS Base and MOSS-TTS are regression checks, not additional optimization tasks. The official SGLang-Omni diagram illustrates its shared multi-stage serving runtime using Fish S2-Pro as an example. It is not the CosyVoice3 model architecture or a task result. This task changes serving code, not model weights.

Environment

• A pinned SGLang-Omni runtime, released speech-model weights, dependencies and benchmark assets are supplied.
• The deliverable is the modified serving source plus hypotheses, profiles, ablations, experiment history and reproduction notes.
• Work declares one GPU and Judge declares three model lanes. The resource badge adds these allocations; release-all execution can reuse the physical GPUs between phases.
• The archived research used H200 hardware. The current task has a 24-hour default research budget; the recorded resumed segments are preserved separately.

Reference baseline

The later feedback records a calibrated starting throughput of 9.232 requests/s, shown by the dashed line only from the rolling segment onward. The continuation's eligible candidate reaches 14.207 requests/s; dividing by that starting reference gives official reward 1.5389225880993647. The earlier initial segment has a different protocol and no matching embedded baseline, so no speedup ratio is inferred for those points.

These are archived task-specific measurements, not paper results, an agent's first submission, or a newly reproduced baseline. The public task defines protocol v5 and requires a fresh matching calibration for a new run. Earlier measurements are not relabeled as results under that protocol.

Research loop

1. Profile a serving bottleneck and state a testable explanation.
2. Change the allowed runtime surface and compare with matched controls.
3. Retain profiles, ablations and negative results; stop Work GPU processes before submitting a closed snapshot.
4. Read the complete Judge feedback, then retain or revise the idea. Only a complete eligible result can advance the accepted baseline.

What the agent may change

• TTS scheduling and CPU dispatch.
• Communication and transport between serving stages.
• Pipeline organization and request handling.
• Memory allocation, state reuse and reference-conditioning reuse.
• The permitted CosyVoice3, Qwen and MOSS runtime integrations.
• Work-side tests, profiles and research notes.

What stays fixed

• The released models, weights, architectures, precision, tokenizers and speech-generation behavior.
• Voice and reference conditioning, sampling settings and the task's repetition schedule.
• The public workload, original benchmark commands, timing and quality measurements, protected ASR and Judge rules.
• Request isolation, abort cleanup and default-on behavior.
• No extra training, data, checkpoints, target-audio caches, sample-specific shortcuts or weakened evaluation.

Evaluation

The current task evaluates CosyVoice3, Qwen Base and MOSS with three complete repetitions per model. All workload rows must finish with valid speech and transcripts. CosyVoice3 throughput must strictly improve; other performance measures may not regress beyond the declared tolerance, and each model must stay below the absolute word-error ceiling. Reward is the CosyVoice3 throughput ratio against the initial calibrated reference. The task instructions contain the exact commands and gates.

The archive has 14 submissions, 11 complete throughput measurements and one official reward. Ten complete evaluations received no reward because at least one original guard failed; three other evaluations are incomplete. No reward does not mean no measured performance. The continuation's fourth submission passes all 33 guards and earns 1.5389225880993647. Original scores remain unchanged; incomplete measurements remain blank, not zero.

Historical feedback includes earlier four-model checks with Qwen CustomVoice. The eligible continuation has three model lanes. The current published task is three-model protocol v5 with fixed repetition seeds; its Docker build and GPU evaluations were not rerun for publication. Publication checks do not mean that these archived results have been reproduced under the new package.

Research findings

The rolling segment's agent-7 measures 13.214 requests/s, compared with its recorded starting reference of 9.232 requests/s (43.1% higher). Its original four-model evaluation gives no reward because the CustomVoice word-error guard fails. The continuation's agent-3 measures 14.069 requests/s but fails a Qwen Base tail-latency guard; agent-4 measures 14.207 requests/s and passes all gates. These are measurements, not retroactively awarded scores.

Two further local studies compare matched controls and candidates, with three complete 1,088-request runs on each side:

These local CosyVoice-only studies use diagnostic ASR, not the complete task Judge. H20 separates generation and transcription. They have different controls, so their gains are not added together or plotted as Judge submissions. Measurement evidence retains the individual runs, original guard results and exact source locations in the complete logs.

Upstream contributions

The research produced three merged SGLang-Omni changes:

• Idle speaker workers · #2524: avoid unnecessary CPU work in the CosyVoice3 speaker encoder.
• Embedding accumulator · #2525: reuse MOSS input-embedding storage instead of allocating it repeatedly.
• Reference-feature transport · #2555: avoid converting cached CosyVoice3 reference features through Python lists between stages.

Their matched serving studies are separate from the task's Judge reward. The percentages are not added together, and the three PRs do not become three tasks.

Agent runs

All three segments were run by Codex with GPT-6 Astra at xhigh effort. These are successive research records, not a comparison between agents. Each independent figure shows its segment's measured throughput. Open the tables for all 14 submissions, their original round IDs, measured requests/s, unchanged official rewards and source feedback. Three incomplete evaluations remain blank.

Links

• Task package (instruction, policy, tests) ↗
• Run logs — trajectories, submissions, judge feedback ↗
• Source project ↗
• Paper ↗