数据集 / zai-org/GLM-TTS

zai-org/GLM-TTS 已完整同步

GLM-TTS: Controllable & Emotion-Expressive Zero-shot TTS

📜 Paper   |   💻 GitHub Repository   |   🛠️ Audio.Z.AI

📖 Model Introduction

GLM-TTS is a high-quality text-to-speech (TTS) synthesis system based on large language models, supporting zero-shot voice cloning and streaming inference. The system adopts a two-stage architecture combining an LLM for speech token generation and a Flow Matching model for waveform synthesis.

By introducing a Multi-Reward Reinforcement Learning framework, GLM-TTS significantly improves the expressiveness of generated speech, achieving more natural emotional control compared to traditional TTS systems.

Key Features

  • Zero-shot Voice Cloning: Clone any speaker's voice with just 3-10 seconds of prompt audio.
  • RL-enhanced Emotion Control: Utilizes a multi-reward reinforcement learning framework (GRPO) to optimize prosody and emotion.
  • High-quality Synthesis: Generates speech comparable to commercial systems with reduced Character Error Rate (CER).
  • Phoneme-level Control: Supports "Hybrid Phoneme + Text" input for precise pronunciation control (e.g., polyphones).
  • Streaming Inference: Supports real-time audio generation suitable for interactive applications.
  • Bilingual Support: Optimized for Chinese and English mixed text.

System Architecture

GLM-TTS follows a two-stage design:

  1. Stage 1 (LLM): A Llama-based model converts input text into speech token sequences.
  2. Stage 2 (Flow Matching): A Flow model converts token sequences into high-quality mel-spectrograms, which are then turned into waveforms by a vocoder.

Reinforcement Learning Alignment

To tackle flat emotional expression, GLM-TTS uses a Group Relative Policy Optimization (GRPO) algorithm with multiple reward functions (Similarity, CER, Emotion, Laughter) to align the LLM's generation strategy.

Evaluation Results

Evaluated on seed-tts-eval. GLM-TTS_RL achieves the lowest Character Error Rate (CER) while maintaining high speaker similarity.

Model CER ↓ SIM ↑ Open-source
Seed-TTS 1.12 79.6 🔒 No
CosyVoice2 1.38 75.7 👐 Yes
F5-TTS 1.53 76.0 👐 Yes
GLM-TTS (Base) 1.03 76.1 👐 Yes
GLM-TTS_RL (Ours) 0.89 76.4 👐 Yes

Quick Start

Installation

git clone [https://github.com/zai-org/GLM-TTS.git](https://github.com/zai-org/GLM-TTS.git)
cd GLM-TTS
pip install -r requirements.txt

Command Line Inference

python glmtts_inference.py \
    --data=example_zh \
    --exp_name=_test \
    --use_cache \
    # --phoneme # Add this flag to enable phoneme capabilities.

Shell Script Inference

bash glmtts_inference.sh

Acknowledgments & Citation

We thank the following open-source projects for their support:

  • CosyVoice - Providing frontend processing framework and high-quality vocoder
  • Llama - Providing basic language model architecture
  • Vocos - Providing high-quality vocoder
  • GRPO-Zero - Reinforcement learning algorithm implementation inspiration

If you fin

17 个文件

浏览文件

数据集版权信息

本数据集的许可证为 MIT License。如有违反相关条款，请联系 WEHUB，我们将及时处理。 查看许可证