NTU and Peking University Collaborate on Open-Source AI Model for Virtual Environments

Researchers from Nanyang Technological University (NTU), Peking University's Wang Xuan Institute of Computer Technology, and the Shanghai Artificial Intelligence Laboratory have jointly released WORLDMEM, an open-source long-memory world model. This breakthrough aims to solve persistent challenges in maintaining consistency within virtual environments, particularly when perspectives shift or time elapses.

Innovative Memory Mechanism

The core innovation of WORLDMEM lies in its advanced memory architecture, which creates a repository of multiple memory units. Each unit stores detailed scene information and state data tied to specific moments in time. This system allows the model to retrieve and reconstruct accurate scenes regardless of viewpoint changes or temporal gaps—a significant improvement over traditional methods limited by short-term context windows.

"What sets WORLDMEM apart is its ability to maintain environmental details over extended periods," explained the research team. "This creates more immersive and coherent virtual experiences."

Dynamic Scene Generation

When generating new scenes, WORLDMEM's system rapidly identifies the most relevant information from its extensive memory library through complex reasoning algorithms. For example:

• If a virtual character moves through an environment and returns to a previous location, the model instantly recalls the original scene configuration
• The system maintains consistency even when objects are moved or environmental conditions change

The model employs conditional diffusion transformers—an architecture that integrates external action signals to enable first-person perspective generation. This allows for natural movement and interaction within virtual spaces.

Continuous Learning Capabilities

WORLDMEM features dynamic updating:

1. New scenes automatically incorporate into the memory library
2. The system continuously refines its understanding of environmental states
3. Scene generation quality improves progressively over time

The research team utilized diffusion forcing technology during training, enabling the model to perform long-term temporal simulations while maintaining scene coherence.

Future Applications

The release of WORLDMEM represents a significant advancement for:

• Virtual reality development
• Gaming environments
• Training simulations
• Architectural visualization

The open-source nature of the project encourages broader adoption and collaborative improvement within the AI community.

Open source address: https://github.com/xizaoqu/WorldMem

Key Points:

✅ Long-term consistency - Maintains scene coherence across time and perspective changes
✅ Dynamic memory - Continuously updates with new environmental information
✅ Open-source availability - Encourages community development and application
✅ Action-responsive - Integrates movement signals for natural interaction