View profile

5G, AR/VR, and AI avatars: A new age of storytelling emerges

5G, AR/VR, and AI avatars: A new age of storytelling emerges

When I was twelve, I used to wait all summer by my bedroom window for an owl to arrive from Hogwarts. I yearned for my invitation into the world of witches, wizards and magic. This was one of my childhood wishes that, unfortunately, did not survive the drudgery of middle school in the Chinese education system. What remained, though, was my desire for something more, both in life and in learning.
Sixteen years later, Niantic, the creator of the popular AR mobile game, Pokémon Go, released another AR mobile game, Harry Potter: Wizards Unite (trailer). With smartphone sensors, location tracking, and AI-powered cameras, Niantic’s new game blurs the line between fantasy and reality, merging J.K. Rowling’s magical universe with ours.
Many believe that AR will be one of the most significant mediums in the next five years. Games like Pokémon Go and Wizards Unite have demonstrated how AR/VR technology is changing the way we play. Beyond gaming, the combination of AR/VR with AI avatars will not only transform how we play, but also how we engage with stories, consume media content, and learn new skills. Let’s dig deeper into this space and look at some recent trends.
How Harry Potter casts an augmented reality spell
How Harry Potter casts an augmented reality spell
People will experience VR/AR on 5G mobile devices
The fifth generation cellular network technology (5G) is a controversial topic because of the recent trade conflicts between the U.S. and China. Not only is 5G crucial to national security, it’s also the key to unlocking the magic of VR/AR. Here are three reasons why VR/AR relies on 5G technology:
  • Response Time: The latency (a.k.a. “time lag”) between a request and the response in 5G networks will decrease about 10x, making VR games faster and providing a smoother gameplay experience.
  • Connectivity and Reliability: Network performance makes or breaks a user’s experience with a VR/AR application. 5G networks offer ubiquitous device usage which means it can handle thousands of devices at a time.
  • Edge Computing: Data storage and computing power for 5G will move from local, physical devices into the edge cloud.
Combined with more powerful cloud computing powered by GPUs in data centers, we will soon have the high bandwidth, low latency, and faster computation necessary to render rich and smooth AR/VR experiences. Google Stadia, the cloud gaming service, is decoupling gaming from both game consoles and game ownership. By streaming game video via the cloud, Stadia enables users to play on their TVs, tablets, and Pixel phones without the need to download the game or purchase a game console.
For AR/VR users, that means separating the virtual or augmented experience from the possession of computers and GPUs. Everyone will be able to run AR applications on the go with their smartphones or AR glasses. Those who can afford a pair of goggles (such as Oculus Go) can consume VR content via streaming.
Interactive and intelligent AI avatars will play a central role in our VR/AR worlds
If 5G and cloud computing are enabling immersive and augmented experiences, then AI avatars will be our virtual guides, connecting the real and the fictional. Just take a look at Magic Leap’s video of Mica:
Magic Leap
Mica experience in progress at the @UnrealEngine-@EpicGames booth No. 349. Watch the way she engages with this #GDC19 attendee while participating in a collaborative art project. She’s present, empowering, bold and definitely NOT your virtual assistant.

Deepfake technology (which we previously covered) is rapidly transforming how we construct realistic-looking virtual characters. Imagine if these characters were powered by AI and could personify interactive avatars in the AR/VR world, or, if we’re being imaginative, historical figures from human history or characters from our favorite shows. By applying AI research developed by DeepMind’s AlphaGo and OpenAI’s Dota 2 to other virtual environments, developers could gain access to tools that make it easier to create intelligent and immersive games in VR/AR.
But to really drive home this virtual experience, AI avatars must be powered by “brains” to help them think more like humans. Several areas of AI have made significant progress in building more humanlike AI avatars for immersive experiences in AR/VR. Voice Recognition and Natural Language Processing are becoming better at helping technologies understand human speech. Computer Vision—facial recognition, eye-tracking, sentiment analysis, gesture recognition, and human activity understanding—make AI smarter at perceiving human behaviors. Text-to-Speech technology is even helping AI characters sound more natural.
Playing a game and seeing a famous Harry Potter character who looks just like an actor in the film is fun, but that cannot compare to the sense of immersion you would feel when Professor Snape’s probing, intense eyes bear into your soul or Voldermort senses your nervousness in a duel. For that kind of experience, AI models that understand human behavior are essential.
With AI avatars, AR/VR will unlock a whole universe of interactive experiences
With AI avatars, AR/VR will unlock a whole universe of interactive experiences
A new medium is arising for story-telling, media consumption, and learning
A novel sometimes feels like a film, capturing the reader’s imagination with the power of words. A film brings stories to the screen as an intense visual feast. A game combines powerful visual sensations with adventure, while relinquishing some control to the user. Bringing AI avatars into the VR/AR world will create a new form of media, one which leverages the strengths of each of these existing media formats. The creation of this new, immersive medium will be highly interdisciplinary, involving a crew of writers, film directors, graphic designers, musicians, subject experts, game developers, and AI engineers.
A new Harry Potter story from J. K. Rowling will not only be written and edited as a book, but as a virtual world with intelligent, interactive characters. Readers will be able to visualize all the details of the book and experience the story, not simply as an observer, but as the protagonist. Stories will become a fusion of words, sensations, moving imagery and active participation.
Visiting the Louvre Museum, tourists will be accompanied by an AI avatar tour guide that escorts them throughout the exhibition. No longer will we need to read non-engaging text descriptions on the walls, but instead, the painters themselves, the painted characters, and even physical objects will come to life through our AR glasses and phones, narrating their stories for us to learn. Learning will become fun and interactive - from museums to the classroom and beyond!
Fable Studio's Lucy is a step closer to a new form of story-telling
Fable Studio's Lucy is a step closer to a new form of story-telling
La Fin
As we imagine the possibilities of this new form of media, progress is already underway. Fable Studio’s Wolves in the Walls, featuring an AI protagonist, Lucy, who can interact with viewers, and the Dalí Museum’s deepfaked version of the surrealist who takes selfies with visitors are two glimpses into this magical future.
In ten years, kids won’t need to sit longingly in their bedrooms, waiting for an owl that will never come. Putting on a pair of goggles will be all it takes to launch them into an entirely new world. They may even keep the goggles on while at school, where intelligent avatars, along with human teachers, educate them about the natural wonders of our world by taking them there in their virtual classroom. That is a future we are looking forward to. One where we use AR/VR and AI as our magical wands to unlock the wonders of the physical world without removing ourselves from reality.
  • Swiftkey’s “puppets” lets users record videos of themselves as cartoon animals that mimic facial expressions (VentureBeat).
  • Can a virtual avatar replace a Hollywood actor? (The Guardian
  • Terminator-lite?: Elon Musk has a plan…to implant a computer in your brain (Wired)
  • 🤓 An Alibaba AI software recently set a world record on a reading comprehension test (MIT Tech Review)
  • Play your cards right: No…not poker?! Meet the new AI system from Facebook that can bluff better than a human (The Verge
  • Privacy please: To master facial recognition technology, companies are using your online pictures and secretly capturing your facial data to train their systems. But what are they doing to protect your privacy? (New York Times)
  • 👀 With 10 billion different combinations an AI was able to solve a Rubik’s cube in one second! Here’s how it did it. (BBC)
  • 👻 Check out this deepfake of Jim Carrey as Jack Nicholson’s character, Jack Torrance, in the infamous typewriter scene from The Shining (Reddit
  • 👩‍⚕️ Amazon’s Alexa is now offering its UK users health advice through a partnership with the National Health Service (BBC
  • Not cool: Google recently revealed that some of its contractors listen to some recordings from its Google Assistant devices (Washington Post
  • 😕 There are objects with shapes that are so weird that robots can’t even figure out how to grasp and hold them. Scientists are now teaching them how (IEEE)
  • 🤯 Meet the Columbia professor who is building self-aware machines (Quanta)
  • 👏 Can prosthetic limbs sense touch? AI may be the answer. (MIT Tech Review)
Thanks for reading!
Subscribe here to Embodied AI! Make sure to forward Embodied AI to your friends and colleagues and tweet about us.
Written by Nahua, edited by Isaac, David, and Moritz. Illustrated by Anny.
Did you enjoy this issue?
Embodied AI - The AI Avatar Newsletter

Embodied AI is the definitive virtual beings newsletter. Sign up for the monthly digest of the latest news, technology, and trends behind AI avatars, virtual beings, and digital humans.

Written with love by Twenty Billion Neurons, an AI startup based in Berlin and Toronto.

In order to unsubscribe, click here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue