Video Translation Pipeline
An automated pipeline to translate videos into target languages while maintaining precise lip-syncing and speaker voice identity.
- Applied OpenAI Whisper for ASR and created a custom syllable-matching duration-alignment algorithm utilizing NLP synonym swapping and beam search.
- Implemented dynamic lip-sync using audio-conditioned U-Nets to manipulate the latent space of pre-trained VAEs.
- Integrated zero-shot voice cloning using acoustic encoders to preserve vocal profiles.