SyncVoice: Towards Video Dubbing with Vision-Augmented Pretrained TTS Model
Training on LRS3 Dataset
Training on Internal Bilingual Dataset
Abstract
Video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness and insufficient audio-visual synchronization, and are often confined to monolingual settings. To address these challenges, we propose SyncVoice, a vision-augmented framework that integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns multi-modal visual features with linguistic representations, enabling temporally synchronized speech synthesis. Experiments on the publicly available LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing, surpassing prior methods in both speech quality and synchronization accuracy. Scaling training to a large-scale Chinese-English bilingual audio-visual dataset further enhances vocal fidelity while preserving synchronization, enabling a unified model for both Chinese and English dubbing.
EN-EN test-set sample (LRS3 for training)
Sample #1
Script: We must also make major structural changes to our state’s jobs agency to protect against outsourcing and cronyism and create good family supporting jobs.
Reference Audio：
|
|
|
|
|
|
Sample #2
Script: His unwavering mission to bring equality and a more civilized society to the United States is something that we must remember through all walks of our lives.
Reference Audio：
|
|
|
|
|
|
Sample #3
Script: And that LGBTQ Nevadans are treated with the respect that they deserve.
Reference Audio：
|
|
|
|
|
|
EN-EN test-set sample (Bilingual version)
Sample #1
Script: We must also make major structural changes to our state’s jobs agency to protect against outsourcing and cronyism and create good family supporting jobs.
Reference Audio：
|
|
|
|
|
|
|
|
Sample #2
Script: His unwavering mission to bring equality and a more civilized society to the United States is something that we must remember through all walks of our lives.
Reference Audio：
|
|
|
|
|
|
|
|
Sample #3
Script: And that LGBTQ Nevadans are treated with the respect that they deserve.
Reference Audio：
|
|
|
|
|
|
|
|
ZH-ZH test-set sample (Bilingual version)
Sample #1
Script: 哎呦，我记得最开始的时候，我我就想我比较喜欢挑战。
Reference Audio：
|
|
|
|
|
|
|
|
Sample #2
Script: 然后在秋天的时候，又要拉家带口的，再飞回到好望角一带，每次的这个飞行里程应该都在一万公里以上。
Reference Audio：
|
|
|
|
|
|
|
|
Sample #3
Script: 接下来呢我们再来关注黑龙江七台河福瑞祥煤矿透水事故救援的最新进展。
Reference Audio：
|
|
|
|
|
|
|
|