Motion Conditioning for Video Diffusion Models
OnlyFlow is a method developed by Obvious to improve control over video diffusion models. Its goal is to generate a video that follows both a text prompt and a desired movement, such as a camera pan or object motion.
Traditional approaches struggle to combine an arbitrary motion with an unrelated prompt, while OnlyFlow tackles this challenge by using optical flow as a conditioning signal.
Optical-Flow-Guided Motion
The pipeline works by first extracting optical flow from an auxiliary video, which describes how pixels move between frames. This motion is encoded through a dedicated motion encoder, and the resulting features are injected into the temporal attention layers of the video diffusion model.
Application as a tool for artistic video creation
“Be Water”, an artwork created using OnlyFlow, was showcased in an immersive space at Artechouse NYC and at Kanvas Dubaï, highlighting the creative potential of this tool. The video artwork is composed of four independently generated videos, each one of them displayed on a wall.
The continuity in the movement used to generate the videos create a global coherence, and the artwork acts both as a proof of work, and a visual demonstration of what can be done using OnlyFlow.
