Captions, an AI-powered video editing startup, has launched a new app called Lipdub for translating clips into 28 languages.
Lipdub is available in the App Store for free and supports several languages, including French, Hindi, Spanish, Italian, Portuguese, Japanese and more. The app even lets users translate videos into Texas slang, Gen Z, pirate and baby talk. The demo video shows that the app can also change lip movement according to the selected target language. However, at times there is a certain lag between the audio and the lip movement.
Users can translate a video of a single person talking during up to one minute and then share it on other social media platforms.
On its website, Captions says that more than 3 million creators have used its eponymous video editing app. The startup claims that it has more than 100,000 daily users. The Captions app offers several AI-powered features around video editing, such as removing “ums” and “ahs,” reducing background noise and enhancing speech. The app also has an “AI Lipdub” feature that can change lip movement in post-production editing if you change the transcript.
Captions was founded in 2021 by Gaurav Misra, who was head of design engineering at Snap. In June, the company secured $25 million in a Series B round led by Kleiner Perkins with participation from Sequoia Capital, Andreessen Horowitz (a16z) and SV Angel. To date, Captions has raised $40 million in funding.
Using translation and AI-dubbing to reach a wider audience is a growing trend. In June, YouTube announced that it is testing an AI-powered tool to let users automatically dub their videos in other languages. The company said it is even working on better lip-syncing. Last month, the company said that it is integrating AI-powered dubbing directly into YouTube Studio for easier access for people looking to convert videos into other languages.
Earlier this month, AI-powered voice-generating platform ElevenLabs released its dubbing tool with support for 29 languages. Rest of World previously reported that dubbing service provider companies are generating millions of dollars by translating content for popular YouTubers like MrBeast.
AI-powered dubbing startups have generated a lot of investor interest, with startups like U.K.-based Papercup and Isreal-based Deepdub raising millions of dollars.