CodecSlime: Temporal Redundancy Compression of Neural Speech Codec via Dynamic Frame Rate

Testset: LibriTTS test-clean

We compare CodecSlime (40Hz dynamic frame rate, built upon BigCodec) with state-of-the-art neural codecs under ≈40Hz or ≈600bps using the LibriTTS test-clean datasets. The results are downsampled to 16 kHz for fair comparison. The demo presents audio samples from CodecSlime and baseline methods, showcasing its performance in temporal compression and reconstruction quality. The numbers in parentheses after model names indicate the encoding bitrate (in kbps) of each model. Specifically, CodecSlime's bitrate is decoupled into two components: content and duration, each explicitly indicated.

Testset: LibriSpeech test-clean

We also evaluated CodecSlime on the LibriSpeech test-clean dataset to further validate its performance. The results are split into two sections based on the quantizer type used: VQ-based and FSQ-based. CodecSlime consistently outperforms the BigCodec baselines in both categories, achieving lower WER and higher intelligibility and quality metrics.

One model for various frame rates at inference time

This experiment evaluates how well a single CodecSlime model generalizes across different inference frame rates. The same CodecSlime model, fine-tuned once at 40 Hz using ScheDFR, is tested under multiple runtime configurations. In contrast, the fixed-rate FFR baselines are individually trained for each specific frame rate (40, 50, 67, and 80 Hz). All models share the same backbone architecture (except for the CNN downsampling rate) and the same quantizer configuration (FSQ with 18225 codes). As shown below, higher frame rates lead to lower WER and higher PESQ. However, CodecSlime consistently outperforms the FFR baseline, demonstrating strong generalization and eliminating the need for retraining at each target rate.

This interactive table compares audio reconstructions from CodecSlime and FFR baselines across varying inference frame rates. The same CodecSlime model is used throughout, with only the frame rate adjusted at test time. In contrast, each FFR variant is separately trained for its target frame rate. We provide 3 utterances from LibriTTS test-clean set, and you can pick any of them through the buttons below.

Testset of unseen languages: MLS subset

MLS subset includes 210 randomly selected dev/test utterances from Multi-lingual LibriSpeech covering major Western languages which are not covered in the training set. The results shows that CodecSlime also generalizes well to unseen languages in both VQ and FSQ settings.

The figure below visually illustrates how the CodecSlime DFR scheduler operates on the 237_126133_000002_000004 utterance from the LibriTTS test-clean set. The top waveform aligns with the forced phoneme sequence, while the black-and-white bar below depicts the model's predicted frame-reduction pattern. Here, the target downsample rate is 2, and the maximum downsample segment length is 4.

As shown in the figure, long silences or sustained vowels are often assigned longer segments, indicating that the learned schedule effectively captures temporal redundancy. However, we also observe that many segments span across phonetic boundaries, suggesting that optimal compression strategies cannot be directly inferred from linguistic structure alone. Instead, they emerge from fine-grained frame-level acoustic similarity, which is nontrivial to design manually. During the Melt stage, the model is exposed to diverse merging patterns and learns empirically effective segmentations—even if they appear counter-intuitive linguistically. This supports the necessity of our two-stage melt-and-cool training and highlights the strength of scheduling directly in the latent space without relying on handcrafted heuristics.

The figure below shows another example 7021_79740_000011_000000 utterance, where similar patterns are observed.

Our Melt-and-Cool recipe adapts an FFR backbone to ScheDFR using a simple two-stage process. Starting from the pretrained fix-frame-rate model (①),

the Melt phase introduces random temporal downsampling on the encoder features (②), where the input is obtained through random cropping of fixed-duration speech segments (1 second). Training begins with no downsampling (i.e., each frame is kept as-is), and as training progresses, the proportion of downsampled segments gradually increases according to a target schedule. The downsampling schemes follow a specified proportion of segment lengths while maintaining random segment ordering, controlled by the Melt scheduler that adjusts the sampling proportion over training (③). This produces a DFR foundation model that supports many downsampling patterns (④).

In the Cool phase, we fine-tune this model with DP-computed optimal schemes (⑤), where the input is also obtained through random cropping of fixed-duration speech segments (1 second). We no longer use random downsampling but instead use only optimal downsample schemes computed via the DP-based scheduler for each training utterance under given target parameters. We freeze the encoder parameters and update only the quantizer and decoder to stabilize learning. To ensure model generalization and stability, each input still has a 30% probability of bypassing downsampling. The final result is a DFR model fine-tuned for the target ScheDFR setup (⑥).

Details of the Melt Manager

To further enhance the model's adaptability to diverse downsampling rates and schemes, the Melt manager samples from a constructed Dirichlet distribution, enabling the proportion vector p to evolve from "easy to hard" scenarios while maintaining randomness. After reaching the preset target distribution p_tgt, the concentration of the Dirichlet distribution is slowly reduced. Additionally, we set a certain probability (typically 50%) for the input utterance to undergo no downsampling, ensuring the model's capability does not deviate excessively.

Algorithm: Random-proportion downsampling sample process of the Melt manager

Symbol legend:

• g: the current training step
• S_p: the number of steps required to reach the target proportion p_tgt (typically 10⁵ steps)
• ρ (rho): the probability of performing no downsampling (typically 0.5, i.e., 50%)
• c: controls how sharply the Dirichlet samples cluster and decays with training (typically 30.0)
• ε (epsilon): a small value to prevent zero entries in d (typically 1.0 × 10^-6)
• π (pi): the training progress factor from 0 to 1
• d: the progress-weighted blend between an "all-ones" (no downsampling) mix and the target mix p_tgt
• α (alpha): the Dirichlet concentration parameter derived from d and c
• p: the vector of per-rate proportions sampled for the current mini-batch

Hyperparameters: Maximum downsampling rate U = 4, target mix p_tgt over rates [1,2,3,4] = [0.1, 0.45, 0.25, 0.2], steps to reach target proportions S_p = 10⁵, concentration control parameter c = 30.0, small constant ε = 1.0 × 10^-6, and skip probability ρ = 0.5.