5G, AR/VR, and AI avatars: A new age of storytelling emerges

When I was twelve, I used to wait all summer by my bedroom window for an owl to arrive from Hogwarts. I yearned for my invitation into the world of witches, wizards and magic. This was one of my childhood wishes that, unfortunately, did not survive the drudgery of middle school in the Chinese education system. What remained, though, was my desire for something more, both in life and in learning.

Sixteen years later, Niantic, the creator of the popular AR mobile game, Pokémon Go, released another AR mobile game, Harry Potter: Wizards Unite (trailer). With smartphone sensors, location tracking, and AI-powered cameras, Niantic’s new game blurs the line between fantasy and reality, merging J.K. Rowling’s magical universe with ours.

Many believe that AR will be one of the most significant mediums in the next five years. Games like Pokémon Go and Wizards Unite have demonstrated how AR/VR technology is changing the way we play. Beyond gaming, the combination of AR/VR with AI avatars will not only transform how we play, but also how we engage with stories, consume media content, and learn new skills. Let’s dig deeper into this space and look at some recent trends.

The fifth generation cellular network technology (5G) is a controversial topic because of the recent trade conflicts between the U.S. and China. Not only is 5G crucial to national security, it’s also the key to unlocking the magic of VR/AR. Here are three reasons why VR/AR relies on 5G technology:

Combined with more powerful cloud computing powered by GPUs in data centers, we will soon have the high bandwidth, low latency, and faster computation necessary to render rich and smooth AR/VR experiences. Google Stadia, the cloud gaming service, is decoupling gaming from both game consoles and game ownership. By streaming game video via the cloud, Stadia enables users to play on their TVs, tablets, and Pixel phones without the need to download the game or purchase a game console.

For AR/VR users, that means separating the virtual or augmented experience from the possession of computers and GPUs. Everyone will be able to run AR applications on the go with their smartphones or AR glasses. Those who can afford a pair of goggles (such as Oculus Go) can consume VR content via streaming.

If 5G and cloud computing are enabling immersive and augmented experiences, then AI avatars will be our virtual guides, connecting the real and the fictional. Just take a look at Magic Leap’s video of Mica:

Deepfake technology (which we previously covered) is rapidly transforming how we construct realistic-looking virtual characters. Imagine if these characters were powered by AI and could personify interactive avatars in the AR/VR world, or, if we’re being imaginative, historical figures from human history or characters from our favorite shows. By applying AI research developed by DeepMind’s AlphaGo and OpenAI’s Dota 2 to other virtual environments, developers could gain access to tools that make it easier to create intelligent and immersive games in VR/AR.

But to really drive home this virtual experience, AI avatars must be powered by “brains” to help them think more like humans. Several areas of AI have made significant progress in building more humanlike AI avatars for immersive experiences in AR/VR. Voice Recognition and Natural Language Processing are becoming better at helping technologies understand human speech. Computer Vision—facial recognition, eye-tracking, sentiment analysis, gesture recognition, and human activity understanding—make AI smarter at perceiving human behaviors. Text-to-Speech technology is even helping AI characters sound more natural.

Playing a game and seeing a famous Harry Potter character who looks just like an actor in the film is fun, but that cannot compare to the sense of immersion you would feel when Professor Snape’s probing, intense eyes bear into your soul or Voldermort senses your nervousness in a duel. For that kind of experience, AI models that understand human behavior are essential.

A novel sometimes feels like a film, capturing the reader’s imagination with the power of words. A film brings stories to the screen as an intense visual feast. A game combines powerful visual sensations with adventure, while relinquishing some control to the user. Bringing AI avatars into the VR/AR world will create a new form of media, one which leverages the strengths of each of these existing media formats. The creation of this new, immersive medium will be highly interdisciplinary, involving a crew of writers, film directors, graphic designers, musicians, subject experts, game developers, and AI engineers.

A new Harry Potter story from J. K. Rowling will not only be written and edited as a book, but as a virtual world with intelligent, interactive characters. Readers will be able to visualize all the details of the book and experience the story, not simply as an observer, but as the protagonist. Stories will become a fusion of words, sensations, moving imagery and active participation.

Visiting the Louvre Museum, tourists will be accompanied by an AI avatar tour guide that escorts them throughout the exhibition. No longer will we need to read non-engaging text descriptions on the walls, but instead, the painters themselves, the painted characters, and even physical objects will come to life through our AR glasses and phones, narrating their stories for us to learn. Learning will become fun and interactive - from museums to the classroom and beyond!

As we imagine the possibilities of this new form of media, progress is already underway. Fable Studio’s Wolves in the Walls, featuring an AI protagonist, Lucy, who can interact with viewers, and the Dalí Museum’s deepfaked version of the surrealist who takes selfies with visitors are two glimpses into this magical future.

In ten years, kids won’t need to sit longingly in their bedrooms, waiting for an owl that will never come. Putting on a pair of goggles will be all it takes to launch them into an entirely new world. They may even keep the goggles on while at school, where intelligent avatars, along with human teachers, educate them about the natural wonders of our world by taking them there in their virtual classroom. That is a future we are looking forward to. One where we use AR/VR and AI as our magical wands to unlock the wonders of the physical world without removing ourselves from reality.

Subscribe here to Embodied AI! Make sure to forward Embodied AI to your friends and colleagues and tweet about us.

Written by Nahua, edited by Isaac, David, and Moritz. Illustrated by Anny.