The photo below taken by VentureBeat’s Dean Takahashi at the first-ever Virtual Beings Summit two weeks ago says it all about this week’s issue of EmbodiedAI: “AI makes a virtual being possible”. Over the past few weeks, several of our readers reached out to me interested in learning more about the intelligence behind AI avatars, such as Millie.

This week, I am excited to take you on a trip deep inside the brain of TwentyBN’s AI avatar, Millie in collaboration with my colleague Guillaume Berger, TwentyBN’s principal AI researcher.

3 video demos are embedded below👇

Millie’s brain is powered by a deep learning model trained on over 3 million proprietary video data collected by TwentyBN. Since our motto is to teach machines to perceive the world like humans, Millie’s brain has two major components, a vision model and an audio model, resembling how our brains process the sights and sounds we experience everyday.

Our vision model is an extension of classical image-based object detectors to videos. Similar to typical object detectors such as YOLOv3, our vision model simultaneously localizes and classifies: it detects objects by putting bounding boxes around them and recognizes the object type of each bounding box (e.g. face, body, etc.). 

Unlike image-based detectors, our model also understands the action being performed by each object. This means that it not only detects and distinguishes a human face from a human body but also determines, for instance, if a face is “smiling” or “talking” or if a body is “jumping” or “dancing”. This is made possible by training our model on video data rather than static images.

Compared to image detectors, which only process two-dimensional images, our model can process the time dimension. On the architecture side, we achieve this by using 3D-convolutional layers instead of 2D-convolutional layers, which enables the neural network to extract the relevant motion features that are needed to recognize the action performed by each object.

This end-to-end approach provides us with a very effective tool for analyzing and understanding human actions. In total, our model is trained to detect over 1,100 types of human activities and hand gestures. Another approach that we experimented with consists of breaking localization and classification into two independent sub-tasks: first, we extract bounding boxes and crop around each object using an object detector; then, we classify the action performed in each crop using a second action recognition model.

Our experience tells us that the end-to-end approach has several benefits over this two-step approach:

Previously relying on cloud solutions, we now have our own at-the-edge audio model for the avatar to understand not only what people do but also what they say. Unlike a typical yet complicated pipeline (see graph below), we are inspired by how humans process speech and went for the most direct and end-to-end approach: speech-to-intent. In this approach, one does not need to listen and transcribe every single word to understand the meaning of another person. A variety of speech-to-intent approaches have been explored (such as this). Similar to the Jasper model, the architecture of our model is fully convolutional.

Roland, TwentyBN’s CEO, favors the speech-to-intent approach:

3 benefits of the speech-to-intent approach include:

As we speak, TwentyBN’s researchers and engineers are working on a new deep learning model called SenseNet, which fuses our vision and audio models into a single audio-visual model for intent classification.

SenseNet is crucial in bringing virtual beings from highly restricted environments into “the wild” (think of a seeing Alexa in a crowded supermarket). AI, just like humans, will need to rely on visual cues, that reveal the origin of a spoken utterance or the context of a scene. Merging vision with audio is a necessary step to achieve high intent classification accuracy in a messy environment.

We hope you have enjoyed this trip inside the brain of a virtual being. Let us know if you have any comments or questions and make sure you share this article with your friends!

Subscribe here to Embodied AI! Make sure to forward Embodied AI to your friends and colleagues and tweet about us. Cheers!

Written by Guillaume and Nahua. Edited by Roland, David, Antoine, and Moritz. Illustrated by Anny.