The evolution of skill discovery in virtual assistants

Dear Readers,

Last Friday, I was enjoying a glass of wine at my friends’ apartment when I caught a glimpse of a newly purchased Google Home that sat in the corner. My curiosity for all embodied AIs took over and I asked them what they use their new tech for. “Besides playing music,” they answered, “We ask it for the time, the weather, and…to turn the lights on and off.”

Like many consumers, my friends have encountered the skill discovery problem in voice-based smart technology. While a smart speaker can do more than report the weather, turn on the lights, and order food—4,200 things for Google Assistant alone—it cannot effectively communicate the countless ways in which it can assist us. Since skill discovery is a crucial element in making virtual assistants more effective and humanlike, we will explore the evolution of skill discovery, covering its challenges, current progress, and how it will continue to develop in the future.

Benedict Evans (Andreessen Horowitz) calls skill discovery in smart speakers a fundamental UX puzzle: the Alexa audio-only interface is convenient, for example, until you expect it to recite its 80,000 skills one by one to a user.

There are 2 factors that make skill discovery especially challenging:

In this article, Ryen W. White (Microsoft Research) lists 8 improvements to virtual speakers so that users can more easily discover new skills for their daily needs:

Most recently, Amazon introduced Alexa Conversations, a Deep Learning approach that allows developers to more effectively improve skill discovery with less effort, fewer lines of code, and less training data. While it is still in “preview”, Alexa Conversations has already generated considerable excitement among developers who build skills for the smart speaker.

Essentially, Alexa Conversations aims to establish a more natural and fluid interaction between Alexa and its users within a single skill. In future releases, the software is expected to bring multiple skills into a single conversation. It also claims to be able to handle ambiguous references, such as, “Are there any Italian restaurants nearby?” (near where?), as well as context preservation when transitioning from one skill to another, such as remembering the location of a certain movie theater when suggesting nearby restaurants.

At Amazon’s re:MARS AI and ML conference in June, Rohit Prasad, VP and head scientist at Alexa, mentioned that Alexa Conversations’ machine learning capabilities can help it predict a customer’s true intention and goal from the direction of the dialogue, thus proactively enabling flow across multiple skills during conversation. If these promises are met, the command-query interaction with Alexa will surely begin to feel more like a natural human interaction.

The progress made by the Alexa team is surely exciting, but conversational AI is not the only area with room for improvement. At Embodied AI we endorse the integration of conversational AI and video understanding into an anthropomorphically embodied assistant brought about through the addition of a camera and screen to the existing speaker interface. 

As our understanding of both natural language processing and computer vision continues to advance, there is little reason to limit virtual assistants to audio. The recent release of Amazon Echo Show 5, Facebook Portal and Google’s Nest Hub Max, all of which come with a camera and a screen, already foreshadow the industry’s movement towards virtual assistants that one day can see and be seen. One could reasonably speculate that the big tech companies are working on visually-enabled and embodied virtual assistants to replace their smart speakers in the near future. It’s a natural extension of their existing product lines.

Benefits of virtual assistants with a camera, screen, and anthropomorphic embodiment include:

Roland Memisevic, TwentyBN’s CEO, envisions a future where our conversations with virtual assistants will, unlike with the current smart speakers, not feel like phone-calls:

Perhaps on another Friday evening, sometime in the near future, I will revisit his friends’ rooftop flat and discover a new virtual assistant, one that is not only well-versed in conversation, but also equipped with eyes for understanding context and identifying needs not captured by words. Perhaps it might even have a digitized human body, becoming a virtual friend, who shares and adds to the lively atmosphere of a mid-summer’s night in Berlin.

As I reach to take a sip from my glass of wine and discover that it is empty, I will hear my friends’ Google Assistant call from the corner, “I think we’re ready for another bottle of the white wine!”

Subscribe here to Embodied AI! Make sure to forward Embodied AI to your friends and colleagues and tweet about us. Cheers!

Written by Nahua, edited by David, Will, Moritz, and Isaac. Illustrated by Anny.