Blog.
Notes on how OpenDub dubs a video: the models it uses, what runs on your device, and what goes to the voice model you choose.
-
How to translate a video into another language
To translate a video you choose the file, the language you want and a voice model. What comes back is the same picture with the speech in the new language, in a clone of the speaker's voice, plus subtitles in both languages. This post covers the steps, the files you get, what stays the same, the 17 languages, and which parts are free, which need your own API key and which need the app on your computer.
-
Keep background music when dubbing a video
Dubbing a video should change the words, not lose the soundtrack. OpenDub separates the voice from the music and effects with Demucs, replaces only the speech, and mixes the new voice back over the music at the original's loudness. This post covers the two modes, replacing the voice or leaving it underneath, and what the separation step needs in the browser and in the app.
-
Voice cloning for multilingual video dubbing
OpenDub dubs a video in a clone of the speaker's own voice. There is nothing to record: the voice is cloned from the densest 6–12 seconds of whole sentences in the original, each line is spoken in the tone it was said in, and every clip is transcribed back and checked. This post covers where the voice comes from, the 17 languages it can speak, which voice models do the cloning and what each one costs.
-
How OpenDub dubs a video in your own voice
OpenDub takes a video in one language and gives it back in another, in a clone of the speaker's voice, with every line timed to the original speech and subtitles burned in. This is what happens in between: eight steps, which of them run on your device, what is sent to the voice model you choose, and how the browser tab, the app on your computer and the command line differ.