Contributor
Last month at San Diego Comic Con, I fulfilled my childhood (and let’s be honest, current) dream of stepping inside a NickToon. In Nickelodeon Entertainment Lab’s Rise of the Teenage Mutant Ninja Turtles VR Interview Experience, I interacted with turtles Mikey and Donnie—voiced live by series talent, Brandon Mychal Smith and Josh Brener, respectively.
I stood against a green screen, selected an avatar (I chose Arnold from Hey Arnold!), put on an Oculus Rift headset, and was transported onto a New York City rooftop. I inhabited a 3D form, but as I looked ahead, I saw two 2D turtles. I interviewed the characters in real time, and their movements perfectly corresponded to their answers–there was no discernable lag.
Given that AI isn’t able to match the conversational speed and nimbleness of real humans just yet, companies like Oculus are experimenting with using live actors in their VR experiences to approximate in-person interactions. However, this was the first time I’ve experienced a live melding of 3D and 2D worlds in VR.
Behind the scenes, this “magic” was made possible by four computers, two puppeteers, two voice actors and a team of eight people running the various stations. (The crew were all wearing ninja bandanas, naturally). Adobe Character Animator, Epic Unreal Engine, and NewTeK NDI were all used to create the interview experience. The pièce de résistance was the bright purple keyboard that was piloted by two team members (one for each turtle). Each key featured a different animated pose, and so the actions of the turtles could be “animated” in real time, akin to playing a symbol piano.
According to Chris Young, SVP of the Entertainment Lab, the impetus of the activation was that his team was looking for an innovative way to help socialize Nickelodeon’s Rise of the Teenage Mutant Ninja Turtles at SDCC and so they came up with the idea of a VR press junket.
Mikey and Donnie in Nickelodeon’s “Rise of the Teenage Mutant Ninja Turtles Live” virtual reality interview experience
The Entertainment Lab has been exploring all aspects of virtual cinema pipelines with animated characters; using full body and facial performance capture, and doing real-time playback in game engines. Per Chris: “Whether it’s streaming live performances into virtual experiences or recorded for a more traditional linear output, these techniques create another tool that we can use to create compelling content for our audience.”
Even without the bandanas, the team had great synergy: “The best part about my team is that everyone brings a different expertise. From artists to engineers, the combination of skills and background made this creatively and technically possible” said Chris.
Although I teased the turtles for being “flat” in my interview, I love that this creative choice was made. It would have been unsettling to suddenly see what is 2D in 3D.
For example, I enjoy the premise and plot of Virtual Rick-Ality, but one of my main criticisms of the game is the connection between assets. Rick and Morty the animated series is 2D, but the game makes them into bulbous characters. This disconnect is off-putting and hurts the immersion of the game.
Although the VR Interview Experience was created specifically for Comic-Con and Nickelodeon doesn’t currently have plans to release a version to the public, they are toying around with other activations for fans.
However, VR isn’t where exploration ends for Nickelodeon, they’re also dabbling in AR. Nick’s new SCREENS UP initiative, first launched at the Nickelodeon Kids Choice Sports Awards, allows viewers to hold their mobile devices (or more likely, parent’s mobile devices) up to the screen while watching to reveal hidden AR content.
This app-driven TV and mobile experience is one of the first in the U.S. designed for kids and families. Expect to see more hybrids like this in the future, as it encourages the watching of live TV as well as app downloads; two desirable outcomes for television content providers.
Overall, the best part of my Ninja Turtles Comic Con demo was how silly and good-natured it all was. With heavy headsets and an onslaught of first person shooters, VR can be kind of serious! This felt social and delightful — the improvisational nature of the conversation allowed me to forget I was in a digital space, I was just hanging out with some dope turtles. It’s a credit to Nickelodeon that they could make something this complex look like child’s play.
The “Rise of the Teenage Mutant Ninja Turtles Live” virtual reality experience was built entirely in-house by Nickelodeon’s Entertainment Lab, which spearheads long-range research and development efforts around new technologies for Nickelodeon and its audience. Chris Young is the SVP of the Entertainment Lab, overseeing its creation. The new Rise of the Teenage Mutant Ninja Turtles series premieres September 17th on Nickelodeon.