Year: 2025
Duration: 4 Months
Project Type: Virtual Production, Storytelling, Cinematic, Unreal Engine, Spatial Audio, Motion Capture
Project Overview
As there is an increasing demand for immersive experience, content creators want to provide new experiences to the audience through different mediums. With the latest technological advancements in immersive media, we have the opportunity to scale it to new heights. So, I want to develop a system and facilitate with a platform for content creators to showcase their contents in a new multi-sensory immersive approach of storytelling. Integrating virtual production LED volumes with immersive spatial audio and player position tracking (via XY coordinates and rotation data) is a cutting-edge approach to creating highly immersive experiences.
"I want to integrate LED volume in some way that supports a creative process for designers, artists, or content creators."
It all begun with this one idea
It all started from the time when I worked on the virtual production LED volume from previous semester. I had a thought about what if we create a virtual presence of the audience just how we do for camera in film virtual production.
Challenges
To build a platform for content creators or any storytellers who want to share their stories in a new immersive medium.
See what new workflows needs to be adapted, what technologies needs to be developed, limitations of these technologies, and find gaps to bridge them in a meaningful way to make it all possible.
Project Objective
To find that people feel more presence when sharing the immersive space with another human who they could see and feel their presence. Does shared experience have an impact on people or it does not matter to the audience?
Explore the integration of spatial audio with LED volume technology.
Does Spatial audio make any difference in each of the person’s experience? Does one experience it different from the other?
Does integrating spatial audio with LED volume elevate the overall visual storytelling experience and the narrative depth and emotional engagement?
To make it easy for the audience to onboard and be ready in the space for the experience without any discomfort of wearing the headphones and onboarding.
Tools and Technologies
Game Engine: Unreal Engine (with nDisplay for LED volumes).
Spatial Audio: Binaural Audio, and Ambisonic Recordings.
Tracking System: Web Socket - Web Serial Communication with UWBs trackers.
Middleware: OSC, NDI, or Syphon for synchronisation.
Outcome & Results
We got a total of 24 people who completed the survey form. All the statistics and data here are real data captured from 24 people who took the experience.
88% felt the onboarding process of getting on the LED volume and wearing the headphones was very easy, where remaining 12% felt it was easy.
75% agree that shared experiences such as sharing the space with another person makes a huge impact and makes you feel real presence and virtual presence more distinguished
63% strongly agree and 37% agree that integrating Spatial Audio has a huge impact in storytelling and elevates the visual experience.
25% and 12.5% of them felt motion capture and spatial audio needs some improvement and refinement respectively.
Considering all the aspects of the experience, 62.5% are very satisfied and 37.5% are satisfied with the experience.
Conclusion
The audience feedback strongly validates the effectiveness of the immersive experience, highlighting that both the technical setup and narrative elements contributed meaningfully to user engagement. The ease of onboarding, with 100% finding it easy or very easy, ensured a smooth entry into the experience. Notably, the presence of another person in the space significantly enhanced the sense of realism and emotional connection, with 75% acknowledging the value of shared presence. The integration of spatial audio proved to be a key storytelling enhancer, with all participants recognizing its impact. While a minority noted areas for refinement in motion capture and audio precision, overall satisfaction was high, with 100% of participants reporting a positive experience. These findings reinforce the potential of shared immersive environments enhanced by spatial audio and LED volume in delivering compelling and emotionally resonant narratives.