Meta announces new AI model that can generate video with sound
Facebook owner Meta announced on Friday the launch of a new AI model called Movie Gen that has the capability to create realistic video and audio content based on user prompts. Meta claims that this new tool can compete with existing media generation tools from leading startups like OpenAI and ElevenLabs.
Samples of Movie Gen's creations showcased by Meta included videos featuring animals swimming and surfing, as well as videos using real photos of individuals engaging in activities such as painting. The tool can also generate background music and sound effects that are synchronized with the video content. Additionally, Movie Gen can be used to edit existing videos.
Video and Audio Capabilities
According to Meta, videos created using Movie Gen can be up to 16 seconds long, while audio clips can extend up to 45 seconds. Blind tests conducted by Meta indicated that the model performs favorably compared to similar tools offered by startups like Runway, OpenAI, ElevenLabs, and Kling.
The announcement of Movie Gen comes at a time when the entertainment industry, particularly Hollywood, is exploring the potential of generative AI video technology. Earlier this year, Microsoft-backed OpenAI unveiled Sora, a product capable of generating movie-like videos in response to text prompts.
Concerns and Considerations
While the entertainment industry sees potential in leveraging AI tools for filmmaking, there are concerns regarding the use of such systems, particularly in relation to copyrighted material. Lawmakers have also raised issues concerning the use of AI-generated deepfakes in elections worldwide.
Meta has mentioned that Movie Gen may not be released for open use by developers, similar to its Llama series of large-language models. The company stated that it is working closely with the entertainment community and content creators to explore the applications of Movie Gen, with plans to integrate it into Meta's products in the coming year.
Dataset and Partnerships
Meta utilized a combination of licensed and publicly available datasets to develop Movie Gen, as outlined in a research paper released by the company. OpenAI has been in discussions with Hollywood stakeholders regarding potential collaborations involving Sora, although no formal agreements have been announced.
Lions Gate Entertainment recently announced a partnership with AI startup Runway, granting access to its film and television library for the training of an AI model. In exchange, the studio and its filmmakers will have the opportunity to enhance their work using the AI model.
