Lumen 2.0
A spatially aware projected-AR platform
for storytelling.

Mixed Reality | Spatial Computing | Projection Mapping

The Intro

In 2019, project Lumen was honoured with the Playable Museum award from Museo Marino Marini, Florence. As part of the award, Museo Marino Marini offered a grant as well as opened up their museum to create a Lumen powered experience for their visitors.

Lumen quote

Through this opportunity, I along with my ex-classmates from CIID: Anoushka Garg, Can Yanardag and Matt Visco started a collective called Senseless.World, through which we are exploring how Lumen can create a storytelling experience for the museum goers. Above Agency, where I currently work is also collaborating with us for the industrial design for Lumen 2.0.

Senseless world

The Process

Museo Marino Marini celebrates the work of the Italian sculptor: Marino Marini. The museum has a wonderful open space that hosts the beautiful works of art from the artist through four different floors.

To understand how the current museum experience feels like, the team visited the museum in Florence in Oct 2019. Through people-centred research, we interviewed experts, historians and curators to understand more about the inception of the museum and the stories behind each of the artworks. We were also able to participate in museum tours conducted by the historian and also shadow another tour which was created for inspiring kids through art.

These activities together informed and generated insights which inspired us to come up with some sacrificial ideas and experiences, and kicked us off to Experience Prototyping. Here, using a wizard of oz style of prototyping, we quickly tested and co-created some of the experiences with both the museum visitors as well as the experts in-house.

We received a lot of valuable feedback through these simple prototypes, which we are currently using to create the final storytelling experience. In this experience, Lumen guides visitors from the entrance of the museum and leads them through a journey that touches upon different artworks. A digital narrative, realised through projections ties together the artworks in the physical space, thereby creating an immersive storytelling experience.

Some of the prototypes:

Lumen experience prototyping

The Technology

Unlike the original Lumen which uses ML based object classification to understand the world around it, Lumen 2.0 is spatially aware and capable of relocalising itself in a physical space. It does so using a technique called V-SLAM (Visual Simultaneous Localisation and Mapping). Using V-SLAM, the onboard tracking and depth cameras can create a point cloud representation of the world and localise itself based on the previously generated map of the space.

Lumen experience prototyping

Lumen experience prototyping

Lumen uses a game engine as a spatial content management system. Through this game engine, Lumen can be tracked through the map of the physical space just like a first-person player in a game. We can then create a virtual world on top of this map and drop in media assets like pictures or videos that will be visible when Lumen is around it.

Lumen can thus augment a physical space by overlaying projected media content on top of it. Similar to a flashlight, Lumen can reveal a virtual world through its projection field. This can help create way-finding experiences or other interactions involving discovering digital artefacts in the physical world.

Lumen projections

Lumen projections

The Design

Victor Johansson, through Above Agency has been helping in realising and prototyping different forms for Lumen. We are currently in the process of trying out some of these forms and playing with the tech stack that goes inside it.

Lumen Machine learning

We believe that through Lumen, museums and other cultural institutions can create a new language of storytelling for their visitors. A language that is inclusive and universal to everyone without having to need a preconceived understanding of art or history.