PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided VLM

Abstract

Inferring physical properties from a single image is fundamentally under-constrained. Attributes such as density, elasticity, and yield stress govern how objects move, yet they are largely invisible in a static frame. Existing physics-aware methods attempt to resolve this ambiguity through task-specific fine-tuning or supervised property estimation, but both strategies struggle to generalize across diverse materials and scenes. We observe that different motions expose complementary physical cues. Building on this observation, we propose PhyMAGIC, a training-free framework that actively probes physical properties by synthesizing targeted motions from a single image. Specifically, PhyMAGIC uses a pretrained image-to-video model to construct motion probes that generate diverse dynamic sequences from the input image. A vision-language model then analyzes these sequences to estimate physical parameters, each accompanied by a confidence score. Parameters with low confidence trigger targeted prompt refinement, which generates additional probe motions to gather complementary evidence. Once all parameters reach sufficient confidence, PhyMAGIC compiles them into a complete physical specification and executes it in a differentiable Material Point Method simulator initialized from 3D Gaussian reconstructions. Experiments on diverse real-world scenes demonstrate that PhyMAGIC achieves stronger text–motion alignment and higher human-rated physical plausibility than state-of-the-art open-source video generators and physics-aware baselines.

Keywords: Dynamic 3D · Physical Simulation · Vision-Language Models · Material Point Method

Motivation

A single image with an ambiguous instruction leaves an object's physics severely under-constrained. PhyMAGIC generates a chain of motion-probe videos that a VLM reasons over, turning coarse, low-confidence guesses into reliable physical parameters — e.g. correcting "elastic" → "rigid" for a toy car once free-fall and collision evidence is observed.

Method Overview

PhyMAGIC pipeline overview — Overview of PhyMAGIC. **(1) Motion Probe Generation** turns the image and an instruction into probe videos; **(2) VLM Physical Reasoning** infers physical parameters with confidence scores and refines the instruction; **(3.1) Hybrid Physical Specification** packages them as Hybrid Physical Parameters (HPP); and **(3.2) MPM Simulation** simulates and renders 3D motion with a differentiable solver — all in a closed loop. · PDF

Perceive · Motion Probe + VLM Reasoning

Infer what the object is made of

Recovers object-level physical parameters from one static image by combining generative video cues with structured VLM reasoning.

Motion Probe Generation — an image-to-video (I2V) model turns the image plus a textual instruction into short probe videos that expose physical cues such as free-fall, collision, and deformation.
VLM Physical Reasoning — a vision-language model estimates material, mass, density, Young's modulus, Poisson ratio, and yield stress in an annotation-free way, each with a confidence score.
Confidence-Driven Refinement — low-confidence parameters trigger an optimized instruction that regenerates more discriminative probe videos, looping until the estimates are reliable.

Simulate · Hybrid Spec + MPM

Simulate how it moves

Turns the inferred properties into physically faithful, renderable 3D dynamics.

Hybrid Physical Specification — inferred values are packaged as Hybrid Physical Parameters (HPP): object-level parameters (material, density, modulus, …) plus simulator-native descriptors (motion, boundary condition, rotation, velocity, surface type).
3D Reconstruction — an image-to-3D (I23D) model lifts the image to 3D Gaussians, converted into a simulation-ready particle representation.
MPM Simulation & Rendering — a differentiable Material Point Method solver advances the particles under the HPP; Gaussian rasterization renders the result, and simulation feedback closes the loop.

Citation

@inproceedings{meng2026phymagic,
  title     = {PhyMAGIC: Physical Motion-Aware Generative Inference
               with Confidence-guided VLM},
  author    = {Meng, Siwei and Luo, Yawei and Liu, Ping},
  booktitle = {ECCV},
  year      = {2026}
}