A mixed reality storytelling platform that
creates collaborative immersion

Mixed Reality | Machine Learning | Projection Mapping

The Intro

Lumen is a Mixed Reality storytelling platform that lets people immerse in alternate realities in their natural space through machine learning and projection mapping technologies. It tries to explore the creation of a new kind of media that takes advantage of the physical world while overlaying a layer of digital fiction on top of it. The main inspiration behind Lumen was imagining a future where people’s imagination are no longer confined to 2D touch screens or headsets, but instead by projecting it outside into the environment and allowing them to interact with the world around them. This therefore lead to the design question:

Lumen design question

All popular AR/VR technologies today that lets you immerse relies on headsets or screens, Lumen challenges this trend and explores how people can feel immersed in their natural space by merging bits with atoms.

The Problem

The tech industry is burgeoning with augmented reality and virtual reality based technologies like the Microsoft Holo lens, Oculus Rift, HTC Vive and many more. But has anyone ever critically thought about the headsets that people wear to experience them?
These AR/VR headsets socially isolate people, you are not able to share the experiences you have while using these headsets with someone else who is not wearing one, and is socially handicapping people. Research has even shown that experiences generated using such technologies on long term dissociate people from reality (paper). Moreover, constraining your movement to a particular location through the tether on the headset only makes it worse.

Lumen headsets

Technology was meant to be designed in a way that it adapts to people, and not in a way where people have to adapt to use such technologies. Do we really want to live in a future where everyone has a headset on their face?

The Process

Lumen was created through a people centred design process by talking, co-creating and collaborating with different people. Starting off with several sacrificial concepts that were used for talking with experts in the realm of AR and MR and to ground them on the same thinking, key interactions were then identified from these story sketches.

Moreover, co-creation and prototyping sessions with a LARP (Live Action Role Playing) storymaster: Troels Andersen, were really helpful for understanding about the elements and experiences that creates immersive stories. As part of research, escape rooms and museums were also studied for understanding the immersion they are capable of creating.

Lumen co-creation

For exploring the interactions, several different concepts such as storytelling with kids, storytelling in art galleries, storytelling in homes, etc were tested with people through functional prototypes. And through these sessions, it was understood that the potential for this kind of tool lies in it being a platform, a storytelling platform that lets people tell their stories through magical interactions with their environment. And that's how Lumen as a platform was born.

The Technology

Lumen uses the yolo darknet machine learning platform to classify objects which is then processed by the onboard algorithm that generates stories on top of those classified objects. Narratives for this platform are designed by storytellers and game designers who will be having access to the background graphical interface that will serve as the story builder and dashboard.

Lumen Machine learning

The hardware consists of a laser projector in combination with a camera and depth sensors that work to create the best projection mapping experience for the user.

Lumen components

In total, three different prototypes were made. Each prototype was an iteration over the earlier one based on the learnings from the testing sessions.

The Design

We all have a deep connection with flashlights, we used to play with them while we were kids and used to tell stories using it. We use a flashlight to find something that we are looking for, to help us find our way in the dark or even to make the invisible visible. Lumen borrows this metaphor of a flashlight to help people create a strong mental model around using it. You can use it just like a flashlight, by pointing and clicking on anything to reveal something hidden.

Lumen flashlight

However, it was also clear while testing that it might not be ergonomic for people to hold Lumen in one hand while trying to interact with something with their other. Further time will be hence spent on making future designs will be sleeker or even adapted to smartphone cases to make it more ergonomic.

The Market

The following graph shows all the current products in the space of mixed reality today that are available in the market. They have been classified based on whether they rely on headsets/screens and on the portability of the device themselves. Lumen lives in an untapped area of mixed reality devices that are portable and which does not confine the user to a headset or screen.

Lumen market

Lumen was created to provoke the tech industry which is currently focussing on only headsets to deliver immersive experiences and push them to think outside them and design a future where people are not constrained by technology.

More Pictures