DeepMind’s new AI generates soundtracks and dialogue for videos

DeepMind, Google’s AI analysis lab, says it’s growing AI tech to generate soundtracks for movies.

In a submit on its official weblog, DeepMind says that it sees the tech, V2A (quick for “video-to-audio”), as an important piece of the AI-generated media puzzle. Whereas loads of orgs together with DeepMind have developed video-generating AI fashions, these fashions can’t create sound results to sync with the movies that they generate.

“Video era fashions are advancing at an unbelievable tempo, however many present methods can solely generate silent output,” DeepMind writes. “V2A expertise [could] change into a promising strategy for bringing generated films to life.”

DeepMind’s V2A tech takes an outline of a soundtrack (e.g. “jellyfish pulsating beneath water, marine life, ocean”) paired with a video to create music, sound results and even dialogue that matches the characters and tone of the video, watermarked by DeepMind’s deepfakes-combatting SynthID expertise. The AI mannequin powering V2A — a diffusion mannequin — was skilled on a mix of sounds and dialogue transcripts in addition to video clips, DeepMind says.

“By coaching on video, audio and the extra annotations, our expertise learns to affiliate particular audio occasions with numerous visible scenes, whereas responding to the data supplied within the annotations or transcripts,” DeepMind writes.

Mum’s the phrase on whether or not any of the coaching knowledge was copyrighted — and whether or not the information’s creators have been knowledgeable of DeepMind’s work. We’ve reached out to DeepMind for clarification and can replace this submit if we hear again.

AI-powered sound-generating instruments aren’t novel. Startup Stability AI launched one simply final week, and ElevenLabs launched one in Might. Nor are fashions to create video sound results. A Microsoft venture can generate speaking and singing movies from a nonetheless picture, and platforms like Pika and GenreX have skilled fashions to take a video and make a finest guess at what music or results are applicable in a given scene.

However DeepMind claims that its V2A tech is exclusive in that it could perceive the uncooked pixels from a video and sync generated sounds with the video robotically, optionally sans description.

V2A isn’t good — and DeepMind acknowledges this. As a result of the underlying mannequin wasn’t skilled on a variety of movies with artifacts or distortions, it doesn’t create significantly high-quality audio for these. And on the whole, the generated audio isn’t tremendous convincing; my colleague Natasha Lomas described it as “a smorgasbord of stereotypical sounds,” and I can’t say I disagree.

For these causes — and to forestall misuse — DeepMind says it gained’t launch the tech to the general public anytime quickly, if ever.

“To ensure our V2A expertise can have a constructive affect on the inventive neighborhood, we’re gathering various views and insights from main creators and filmmakers, and utilizing this helpful suggestions to tell our ongoing analysis and improvement,” DeepMind writes. “Earlier than we contemplate opening entry to it to the broader public, our V2A expertise will bear rigorous security assessments and testing.”

DeepMind pitches its V2A expertise as an particularly great tool for archivists and folk working with historic footage. However, as I wrote in a bit this morning, generative AI alongside these traces additionally threatens to upend the movie and TV business. It’ll take some critically sturdy labor protections to make sure that generative media instruments don’t remove jobs — or, because the case could also be, whole professions.