Introduction
Ako is a conversational augmented-reality AI assistant that helps a player learn a table-top game. The project explored what users actually want from a virtual conversational agent while they’re engaged in the physical world. We were a team of four: two programmers, one artist, and me, the designer and prototyper.
The unknowns going in
Behavior
- Will people talk to a character to learn, or just grab the rulebook?
Technical
- Can passthrough AR + speech + hand-tracking actually hold together, live?
- We used off-the-shelf components for each feature, since these technologies were not built-in or refined at the time.
Character
- Can a personality stay charming when it inevitably misunderstands you?
Design goals
- Make learning the game feel magical.
- Preserve the fun of the physical, tactile game.
- Give the agent a personality worth talking to.
From paper system to integrated prototype
As producer and designer, I managed the four-person production pipeline while owning the interaction model, UI and wireframes, dialog system and character personality. The functional prototype combined speech processing through Amazon Polly with passthrough AR and hand-tracked interaction around a physical table-top game.
Paper and Wizard-of-Oz prototypes let us validate behavior before every technical component was ready. I then worked closely with the programmers and artist to translate those findings into buildable states, animations and recovery behaviors—keeping the experience coherent even when speech or tracking confidence was imperfect.
Prototyping the dialog
While the programmers integrated the tech, the artist and I ran paper prototypes. We designed several dialog systems and iterated to find the sweet spot that helped players understand the game being taught.
Designing the dialog system came with real tension: we wanted to be helpful and teach the game, with visuals and animations closely linked to the tutorial. But we also needed a loop that maintained the user’s interest. Along the way we iterated on table-top games and character styles until we found something players loved to interact with.
What playtesting taught us
Our prototypes and playtests overturned a lot of my assumptions:
- I assumed Ako should always be listening to help. But Ako often picked up dialog not directed at him, causing confusion. I changed the behavior so that when Ako figures the dialog isn’t relevant, he simply gives a neutral reaction.
- We set out to teach all the rules upfront, like a rulebook. We were surprised how many players just wanted to pick up the cards and play.
- People arranged cards in whatever way made sense to them. Humans don’t follow one pattern; there’s no ideal way when freedom is afforded. The design had to adapt to this dynamic behavior.
Presence invites interaction
Even though users could talk to Ako, his mere presence made them want to touch and interact with him. We had hand-tracking for cards, but received overwhelming feedback asking to play with Ako directly.
Designing an agent lovable to the user meant users also expected reciprocation. Version 2 of Ako let users interact with him. The trade-off was more hand- and character-tracking challenges, but the experience of interaction overshadowed the drawbacks, so the team kept it.
Being helpful, not pushy
Across many iterations, the artist and I collaborated on showing hints and finding the sweet spot for capturing attention: helpful, but never pushy.
Many users wanted to skip the tutorial. So when players made mistakes early in the game, Ako stepped in with an extra ounce of support, especially when introducing new levels and features, so players still felt supported throughout. We also found that when users concentrated on the physical world, they forgot Ako was there. That led us to add a smidge of extra vocal and visual cues to signal help was available.
Outcomes
- Users overwhelmingly preferred conversing with Ako over a rulebook.
- Visually pleasing elements were always read as an affordance; let them be interactive.
- An embodied AI brings delight and presence when done right.
- Where the system wasn’t robust enough, I designed it to fail gracefully rather than break immersion.
What I took away
- Wizard-of-Oz prototypes delivered discoveries early, saving the team time and giving us insight to design before the tech was ready.
- Testing early and often was our gold mine; it challenged our assumptions and taught us to design for user needs and delight.
- When the prototype hit a situation beyond scope, I designed Ako to honestly explain the limitation and keep the game moving. Knowing our gaps helped us design for graceful failure and scope future work.