Kalevala. Episodes.
Interactive VR performance
Golden Mask Theater Award 2020
nomination Experiment
Description
“Kalevala. Episodes” is based on the stories of the Finnish-Karelian mythological epos and folklore.
This is a 40 minutes fantasy immersive trip into the mysterious worlds of Kalevala heroes, ancient culture, and atmosphere made using modern vision and sound technology, 3D 360 video, volumetric capture, and interactive scenes.
Kalevala is an utterly mystical, surrealistic space of the ancient epos, where Christian culture and the spirit of paganism are mixed and coexist so far.
Using electronic sound design and virtual tools we create a modern VR story of Kalevala.
The viewer is immersed not only in an entirely new space and culture but also in a new kind of virtual experience and storytelling.
We use our own technology to embed stereoscopic video into a 3D environment. Due to this, we can see real actors in virtual scenery.
This interactive VR project was the first virtual performance ever to be nominated for the highest theatrical award, the Golden Mask, Russia, 2020.
“Kalevala. Episodes” is an interactive VR experience available on Meta Quest (link), Oculus Rift, and HTC Vive. We use the Leap Motion controller to track hand movements. It allows the viewer to move around in virtual space and interact with objects, feeling immersed in the story, and surrounded by events and characters.
We also made an abridged version – a 25-minute 3D360 video taken from the interactive project.
Available on Meta Quest TV.
Credits
Director – Oleg Nikolaenko
Senior VR developer – Daniil Bakalin
Producer – Elli Niaria
Producer (Karelia) – Natalya Shamray
Sound design, music – Grigory Arkhipov
Consultant – Olga Ogneva
Original soundtrack – Alexander Leonov, Olga Gajdamak (Yarga Sound System), Arto Rinne
Supported by:
KONE foundation
National Theater of Karelia
TINFO
InterLab ry
Festivals and official screenings
Golden Mask Theater Award 2020, nomination Experiment, Moscow, Russia
Open Frame Award, GoEast Film Festival 2019, Wiesbaden, Germany
International Theater Festival LIFT, Petrozavodsk, Russia, 2018 – Special award
Match Up 2018, Creathon, Metropolia University of Applied Sciences, Helsinki, Finland
TheaterVR Days 2019, St. Petersburg, Russia,
Kalevala days 2019, The Finnish Science Center Heureka, Vantaa, Finland
Specifications
Interactive:
LBE (6DOF) Oculus Rift, HTC Vive
Duration 40 min
Video 3D360:
Duration 25 min
English/Finnish
Making Of
In 2018, we released our first interactive VR performance with real volumetric actors, “Kalevala.Episodes.” Unfortunately, due to its experimental format, a large amount of 3D video involved (130 GB), and the complicated setup and calibration required for showings, we were unable to package the project and submit it to VR festivals. As a result, this VR show remained relatively obscure within the VR community. We hope our story will be of interest to creators of VR Storytelling and everyone interested in Virtual Reality.
After spending a year learning about VR tools and features, as theater makers, we set ourselves the task of creating a 6DoF virtual performance that incorporates high-quality 3D video footage of real actors (not 3D models, even if they are animated with motion capture) into a virtual environment, allowing the audience to move and interact.
The technology of volumetric video was quite expensive in 2018, and even now, it is not widely available, and the quality is still not good enough. Upon closer examination, we found that it works for illustrating dreams and memories using glitchy, deteriorating figures.
However, for a theatre performance, it was not necessary for the audience to walk around the actors. The audience could move within a small range relative to the actors, approach them, closely observe them, and interact with objects in the space.
Our first project is based on the Finnish-Karelian epic poem “Kalevala”. This incredibly powerful and impressive work combines various styles and genres, ranging from cosmogony and powerful tragedy to total surrealism, grotesque, and comedy. The worlds and events depicted in “Kalevala” were perfectly suited to be recreated in VR environments.
We developed the script by selecting some of the most visually impressive and multi-genre stories that could be realized using the VR tools and technologies available at the time. Our goal was not to create a VR movie with actors placed in a virtual environment; rather, we aimed to add a sense of presence and viewer immersion by enabling interaction with objects and movement around the VR space. To achieve interactivity in this project, it had to be 6DoF and rendered in real-time using Unity.
We used stereoscopic footage with actors shot with a stereo camera (GoPro3) on a chromakey. Then we sent the videos (4K 3D video with alpha channel) into specially designed stereo shaders and embedded them into the 3D space in Unity. If the viewer moved left and right, our video shader automatically rotated in space, ensuring no geometric distortion.
After shooting and processing the first tests, we got amazing results!
Before we started shooting, we developed a lot of sketches of the scenes. This allowed us to plan the movements of the actors and the angles at which they should be filmed.
Thanks to the interactive capabilities of 6Dof, the viewer can move within scenes, pass through a portal from one environment to another world, play with stones flying in weightlessness, approach objects and actors, and examine them very closely. Or, for example, the actors walk around the viewer, inviting him to turn 180 degrees and directing his attention, to see the next scene behind.
In this performance, we used both 3D-modeled scenes with embedded actors and scenes filmed in 3D360 format in real locations with actors. The filming took place in historical locations related to the events of Kalevala, specifically on the ancient Girvas volcano (2 billion years old) in Karelia.
I would like to highlight the exceptional modern electronic soundtrack created by Karelian musicians, Yarga Sound System, which is based on traditional melodies and songs. It is an integral part of our virtual experience, providing emotional and artistic depth.
We also wanted to show some mythical characters by creating 3D video using point cloud technology. To implement this, we captured the movements of the actors in a three-dimensional format using Kinect 2.
Some of the 3D models we used in the scenes were made by photogrammetry from real objects we found. For example, the ship on which the characters of Kalevala traveled by sea.
Also, using a depth map, we made three-dimensional models of icebergs from ordinary photos.
In the first scene we wanted to create a seamless transition for the viewer from reality to virtual space. At that time, there were no MR headsets with high-quality cameras (at least we didn’t have them), so we achieved this effect by shooting short POV footage on a VR camera for each viewer before each show in a specific space. Wearing a headset (we used Oculus Rift and Vive), the viewer saw the same space (but captured on video footage that we had uploaded to a prepared scene in Unity beforehand). A person would appear in front of the viewer, ask the viewer to look left and right, and ask, “Is everything okay? Are you comfortable?” And then a “Wow!” would happen. All viewers didn’t hesitate to answer and talk to the character in the video, thinking they could see him through the headset! And the show had already started. Then virtual objects began to appear in this real scene, the light changed, then costumed actors appeared on stereo shaders and walked around the viewer, reality smoothly dissolved, and the viewer gently found himself already in the virtual space of the performance.
You can find a 26-minute video version of our project in 3d360 format (without interactive features), and watch it on the Meta Quest headset on Meta TV.
Kalevala.Episodes
interactive VR performance
Golden Mask
Theater Award 2020
nomination Experiment
Description
“Kalevala. Episodes” is based on the stories of the Finnish-Karelian mythological epos and folklore.
This is a 40 minutes fantasy immersive trip into the mysterious worlds of Kalevala heroes, ancient culture, and atmosphere made using modern vision and sound technology, 3D 360 video, volumetric capture, and interactive scenes.
Kalevala is an utterly mystical, surrealistic space of the ancient epos, where Christian culture and the spirit of paganism are mixed and coexist so far.
Using electronic sound design and virtual tools we create a modern VR story of Kalevala.
The viewer is immersed not only in an entirely new space and culture but also in a new kind of virtual experience and storytelling.
We use our own technology to embed stereoscopic video into a 3D environment. Due to this, we can see real actors in virtual scenery.
This interactive VR project was the first virtual performance ever to be nominated for the highest theatrical award, the Golden Mask, Russia, 2020.
“Kalevala. Episodes” is an interactive VR experience available on Meta Quest (link), Oculus Rift, and HTC Vive. We use the Leap Motion controller to track hand movements. It allows the viewer to move around in virtual space and interact with objects, feeling immersed in the story, and surrounded by events and characters.
We also made an abridged version – a 25-minute 3D360 video taken from the interactive project.
Available on Meta Quest TV.
Festivals and official screenings
Golden Mask Theater Award 2020, nomination Experiment, Moscow, Russia
Open Frame Award, GoEast Film Festival 2019, Wiesbaden, Germany
International Theater Festival LIFT, Petrozavodsk, Russia, 2018 – Special award
Match Up 2018, Creathon, Metropolia University of Applied Sciences, Helsinki, Finland
TheaterVR Days 2019, St. Petersburg, Russia,
Kalevala days 2019, The Finnish Science Center Heureka, Vantaa, Finland
Credits
Director – Oleg Nikolaenko
Senior VR developer – Daniil Bakalin
Producer – Elli Niaria
Producer (Karelia) – Natalya Shamray
Sound design, music – Grigory Arkhipov
Consultant – Olga Ogneva
Original soundtrack – Alexander Leonov, Olga Gajdamak (Yarga Sound System), Arto Rinne
Supported by:
KONE foundation
National Theater of Karelia
TINFO
InterLab ry
Specifications
Interactive:
6DoF Oculus Rift, Quest Link, HTC Vive
Duration 40 min
Video 3D360:
Oculus Go Samsung GearVR Pico 2
Duration 25 min
Finnish/English
Making Of
In 2018, we released our first interactive VR performance with real volumetric actors, “Kalevala.Episodes.” Unfortunately, due to its experimental format, a large amount of 3D video involved (130 GB), and the complicated setup and calibration required for showings, we were unable to package the project and submit it to VR festivals. As a result, this VR show remained relatively obscure within the VR community. We hope our story will be of interest to creators of VR Storytelling and everyone interested in Virtual Reality.
After spending a year learning about VR tools and features, as theater makers, we set ourselves the task of creating a 6DoF virtual performance that incorporates high-quality 3D video footage of real actors (not 3D models, even if they are animated with motion capture) into a virtual environment, allowing the audience to move and interact.
The technology of volumetric video was quite expensive in 2018, and even now, it is not widely available, and the quality is still not good enough. Upon closer examination, we found that it works for illustrating dreams and memories using glitchy, deteriorating figures.
However, for a theatre performance, it was not necessary for the audience to walk around the actors. The audience could move within a small range relative to the actors, approach them, closely observe them, and interact with objects in the space.
Our first project is based on the Finnish-Karelian epic poem “Kalevala”. This incredibly powerful and impressive work combines various styles and genres, ranging from cosmogony and powerful tragedy to total surrealism, grotesque, and comedy. The worlds and events depicted in “Kalevala” were perfectly suited to be recreated in VR environments.
We developed the script by selecting some of the most visually impressive and multi-genre stories that could be realized using the VR tools and technologies available at the time. Our goal was not to create a VR movie with actors placed in a virtual environment; rather, we aimed to add a sense of presence and viewer immersion by enabling interaction with objects and movement around the VR space. To achieve interactivity in this project, it had to be 6DoF and rendered in real-time using Unity.
We used stereoscopic footage with actors shot with a stereo camera (GoPro3) on a chromakey. Then we sent the videos (4K 3D video with alpha channel) into specially designed stereo shaders and embedded them into the 3D space in Unity. If the viewer moved left and right, our video shader automatically rotated in space, ensuring no geometric distortion.
After shooting and processing the first tests, we got amazing results!
Before we started shooting, we developed a lot of sketches of the scenes. This allowed us to plan the movements of the actors and the angles at which they should be filmed.
Thanks to the interactive capabilities of 6Dof, the viewer can move within scenes, pass through a portal from one environment to another world, play with stones flying in weightlessness, approach objects and actors, and examine them very closely. Or, for example, the actors walk around the viewer, inviting him to turn 180 degrees and directing his attention, to see the next scene behind.
In this performance, we used both 3D-modeled scenes with embedded actors and scenes filmed in 3D360 format in real locations with actors. The filming took place in historical locations related to the events of Kalevala, specifically on the ancient Girvas volcano (2 billion years old) in Karelia.
I would like to highlight the exceptional modern electronic soundtrack created by Karelian musicians, Yarga Sound System, which is based on traditional melodies and songs. It is an integral part of our virtual experience, providing emotional and artistic depth.
We also wanted to show some mythical characters by creating 3D video using point cloud technology. To implement this, we captured the movements of the actors in a three-dimensional format using Kinect 2.
Some of the 3D models we used in the scenes were made by photogrammetry from real objects we found. For example, the ship on which the characters of Kalevala traveled by sea. Also, using a depth map, we made three-dimensional models of icebergs from ordinary photos.
In the first scene we wanted to create a seamless transition for the viewer from reality to virtual space. At that time, there were no MR headsets with high-quality cameras (at least we didn’t have them), so we achieved this effect by shooting short POV footage on a VR camera for each viewer before each show in a specific space. Wearing a headset (we used Oculus Rift and Vive), the viewer saw the same space (but captured on video footage that we had uploaded to a prepared scene in Unity beforehand). A person would appear in front of the viewer, ask the viewer to look left and right, and ask, “Is everything okay? Are you comfortable?” And then a “Wow!” would happen. All viewers didn’t hesitate to answer and talk to the character in the video, thinking they could see him through the headset! And the show had already started. Then virtual objects began to appear in this real scene, the light changed, then costumed actors appeared on stereo shaders and walked around the viewer, reality smoothly dissolved, and the viewer gently found himself already in the virtual space of the performance.
You can find a 26-minute video version of our project in 3d360 format (without interactive features), and watch it on the Meta Quest headset on Meta TV.
