Lumiere: Google’s Realistic AI Text-to-Video Generator

Google researchers have unveiled Lumiere, a new time-and-space diffusion model that can transform text or images into realistic AI-generated videos. Lumiere uses a Space-Time U-Net architecture to create videos with “realistic, diverse, and coherent motion.” Unlike other AI video generators, Lumiere can instantly generate the entire duration of a video in a single pass. By utilizing spatial and temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, Lumiere can process textual descriptions or still images with prompts to produce dynamic videos.

Many users have compared Lumiere to ChatGPT, as it offers text and image to video generation, stylization, editing, animation, and more. While existing AI video generators like Pika and Runway already exist, Lumiere’s unique approach to temporal data dimension in video generation sets it apart. A student researcher involved in developing Lumiere, Hila Chefer, showcased its capabilities on the social media platform X, prompting enthusiastic responses from users who called it an “incredible breakthrough” and “state-of-the-art.”

Lumiere was trained on a dataset comprising 30 million videos and text captions. It can generate 80 frames at 16 frames per second. The researchers did not disclose the source of the training data, which has raised concerns about copyright infringement. Numerous copyright infringement-related lawsuits have been filed against developers of generative AI models for allegedly misusing copyrighted content during training. A prominent example is The New York Times’ lawsuit against Microsoft and OpenAI, the creators of ChatGPT, accusing them of “illegally” sourcing their work for training purposes.