NTU and Shanghai AI Lab Pioneer PhysX-3D for Realistic AI-Generated 3D Models

While AI-generated 3D models have made strides in visual realism, they often lack the physical properties that define real-world objects. A collaborative effort between Nanyang Technological University (NTU) and Shanghai AI Lab seeks to bridge this gap with the launch of PhysX-3D, a project designed to embed physical authenticity into AI-generated 3D content.

The Problem: 'Plastic-Like' Virtual Worlds

Current AI-generated 3D models excel in appearance but falter in simulating real-world physics. Objects lack weight, material properties, and functional dynamics—critical for applications like robotic grasping, physical simulation, and embodied intelligence.

The Solution: Five Core Physical Dimensions

The PhysX-3D team identified five foundational questions to guide realistic 3D generation:

1. Absolute Size: Ensuring objects have correct dimensions (e.g., a wardrobe vs. a model).
2. Material Properties: Defining whether an object is made of glass, metal, or sponge.
3. Functional Affordance: Identifying how an object is used (e.g., a chair's seat for sitting).
4. Kinematic Characteristics: Detailing movable parts and motion ranges.
5. Functional Description: Explaining an object's purpose in natural language.

Building PhysXNet: A Groundbreaking Dataset

To train AI systems, the team created PhysXNet, the first dataset annotating all five physical dimensions. It includes over 26,000 meticulously labeled 3D objects, with an extended version, PhysXNet-XL, boasting 6 million models. The annotation process combined AI automation (GPT-4o) and human expertise for precision.

The PhysXGen Framework: Merging Geometry and Physics

The team developed PhysXGen, a dual-branch framework:

• Structural Branch: Handles geometric generation (shape, texture).
• Physical Branch: Learns and predicts physical properties.

The system outperformed traditional methods, reducing material prediction errors by 64% and affordance errors by 72%. For example, it accurately modeled a faucet's rotational joint and an office chair's fabric material.

Implications for Industry and Research

PhysX-3D shifts focus from mere visual fidelity to physically grounded modeling, with potential impacts on:

• Robotics: More accurate object interactions.
• Autonomous Driving: Better simulation environments.
• Virtual Reality: Enhanced realism for immersive experiences.

Key Points:

• PhysX-3D introduces physical realism to AI-generated 3D models.
• The project addresses five core dimensions: size, material, function, kinematics, and description.
• PhysXNet provides the first comprehensive dataset for physical annotations.
• PhysXGen combines geometric and physical modeling for superior results.
• Applications span robotics, VR, and autonomous systems.

Paper link: https://arxiv.org/pdf/2507.12465