Track the Noise, Move the World:3D-Grounded Motion-Consistent Noise for Controllable Video Generation

Summary · qwen2.5:32b

UniCaMo introduces a unified framework enabling precise control over object and camera motions in video synthesis by manipulating the input noise of diffusion models, maintaining consistency across movements and viewpoints without altering the model architecture. This method achieves state-of-the-art results in motion controllability and video quality on standard benchmarks using lightweight LoRA fine-tuning on large pretrained models like Wan 2.1 (14B). Specifically, UniCaMo uses sparse 3D point tracks to guide noise warping along object trajectories while a spherical noise representation ensures consistency for newly revealed scene areas under camera motion.

Suggested post angle

Discover a new approach to generating realistic videos for game development and beyond!

Excerpt

arXiv:2607.02798v1 Announce Type: cross Abstract: Modern image-and-text-to-video diffusion models can synthesize highly realistic videos by iteratively denoising an initial Gaussian noise tensor conditioned on reference image and text inputs. However, existing approaches still lack precise and unified controllability over both object motion and camera motion within a single generation process. We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints by directly constructing the input noise of the diffusion model. Specifically, UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames. Sparse 3D point tracks are used to warp the Gaussian noise of the reference frame along desired object trajectories, while a virtual spherical noise representation provides globally consistent noise values for newly revealed scene regions under camera motion. By combining local track-guided noise warping with global sphere-based noise sampling, UniCaMo maintains geometric and temporal consistency under both object movement and viewpoint changes. Because UniCaMo modifies only the input noise, it requires no auxiliary adapters, control branches, or architectural changes to the underlying video diffusion model. With lightweight LoRA fine-tuning on large pretrained video diffusion models, including Wan 2.1 (14B), UniCaMo achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.