GLA-Grad: A Griffin-Lim Extended Waveform Generation Diffusion Model

Abstract

Diffusion models are receiving a growing interest for a variety of signal generation tasks such as speech or music synthesis. WaveGrad, for example, is a successful diffusion model that conditionally uses the mel spectrogram to guide a diffusion process for the generation of high-fidelity audio. However, such models face important challenges concerning the noise diffusion process for training and inference, and they have difficulty generating high-quality speech for speakers that were not seen during training. With the aim of minimizing the conditioning error and increasing the efficiency of the noise diffusion process, we propose in this paper a new scheme called GLA-Grad, which consists in introducing a phase recovery algorithm such as the Griffin-Lim algorithm (GLA) at each step of the regular diffusion process. Furthermore, it can be directly applied to an already-trained waveform generation model, without additional training or fine-tuning. We show that our algorithm outperforms state-of-the-art diffusion models for speech generation, especially when generating speech for a previously unseen target speaker.

Method

Training-free, inference-time correction

GLA-Grad generates a speech waveform from a mel spectrogram by adding Griffin-Lim phase recovery at inference time. Here, training-free means the extension requires no additional model training or fine-tuning; it still uses pretrained WaveGrad weights.

1. Estimate the target magnitude spectrogram with the pseudo-inverse of the mel filterbank.
2. Run a WaveGrad reverse diffusion update, then correct the current waveform using Griffin-Lim iterations guided by that magnitude estimate.
3. Apply this correction during the early sampling steps, then finish with ordinary WaveGrad updates.

Conditioning consistency

Conditioning consistency describes agreement between the generated waveform's magnitude spectrogram and the target magnitude estimated from the input mel spectrogram. GLA-Grad uses Griffin-Lim correction to reduce this mismatch during early sampling. The target magnitude is a pseudo-inverse estimate, so this constraint does not guarantee exact recovery of the original waveform. See the conditioning constraint in Section 3.2.

Configuration evaluated in the paper: six reverse diffusion steps (WG-6), with GLA correction on the first three steps only and 32 GLA iterations per corrected step. The remaining three steps use the original diffusion update. See Section 3.2 and Section 4.2 for the algorithm and parameters.

Experimental results: unseen-speaker generalization

These are the published results from Section 4 of arXiv:2402.15516v1. In the LJ Speech → VCTK experiment, GLA-Grad achieves PESQ 2.734 versus WaveGrad's 2.076 using WG-6. GLA-Grad has the highest PESQ and STOI among the listed methods in both unseen-speaker setups; SpecGrad has the lowest WARP-Q in those setups. For the same-speaker LJ Speech test, GLA-Grad scores below WaveGrad on all three metrics.

Reading the tables: values are mean ± standard deviation as reported in the paper. Higher PESQ (Perceptual Evaluation of Speech Quality) and STOI (Short-Time Objective Intelligibility) are better; lower WARP-Q is better. WARP-Q measures speech quality using a time-warped comparison with reference audio. Shading identifies the proposed method, not the best score.

Evaluation conditions: the paper computes conditioning mel spectrograms from reference recordings, using 22,050 Hz audio, a 2,048-point STFT, a 1,200-sample Hann window, a 300-sample hop and 128 mel bands. GLA-Grad, WaveGrad and SpecGrad use WG-6; WaveGrad-50 uses WG-50. The standalone Griffin-Lim baseline uses 1,000 iterations. See datasets and metrics and baseline configurations.

Limitations

• Performance depends on the task and metric. These experiments support generalization to unseen speakers, with the same-speaker and WARP-Q qualifications described above. They do not establish superiority over every vocoder or on every dataset.
• Correction is limited to early steps. Applying GLA throughout sampling can damage an already-refined waveform. The number of corrected steps and GLA iterations affects quality.
• No extra training does not mean zero overhead. A trained WaveGrad model is still required; magnitude estimation and phase recovery add inference cost relative to six-step WaveGrad.
• The input is a reference-derived mel spectrogram. The reported scores evaluate waveform reconstruction and speaker generalization, not an end-to-end text-to-speech system with predicted mel inputs.

Sources: method configuration, experimental discussion and inference complexity.

Audio examples

Compare the reference recording with GLA-Grad and WaveGrad for each training → evaluation setup. E1 denotes LJ Speech → LJ Speech, E2 denotes LJ Speech → VCTK, and E3 denotes VCTK → VCTK. The suffix identifies the second set of archived examples.

About these archived samples: the original demo mentioned WG-6 and a searched schedule, but the schedule assigned to each set can no longer be confirmed. Original dataset utterance IDs and verified transcripts are also unavailable. Treat these as listening examples; use the published tables above for comparisons with a known inference configuration.

All clips are mono. Generated audio is stored at 22,050 Hz; the VCTK reference clips are stored at 48,000 Hz. Each link lists the actual file's duration and sample rate.

Citation

Haocheng Liu, Teysir Baoueb, Mathieu Fontaine, Jonathan Le Roux, and Gaël Richard. “GLA-Grad: A Griffin-Lim Extended Waveform Generation Diffusion Model.” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024, pp. 11611–11615. DOI: 10.1109/ICASSP48485.2024.10446058.

Preprint: arXiv:2402.15516 (arXiv DOI). Conference bibliographic details: MERL publication record.

Download BibTeX (.bib)

@inproceedings{liu2024glagrad,
  author    = {Liu, Haocheng and Baoueb, Teysir and Fontaine, Mathieu and {Le Roux}, Jonathan and Richard, Ga{\"e}l},
  title     = {{GLA-Grad}: A {Griffin-Lim} Extended Waveform Generation Diffusion Model},
  booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
  year      = {2024},
  pages     = {11611--11615},
  publisher = {IEEE},
  doi       = {10.1109/ICASSP48485.2024.10446058},
  url       = {https://doi.org/10.1109/ICASSP48485.2024.10446058}
}