2019 – Gesture tutorial app for augmented reality


In my first project at Crunchfish, I explored how using gesture recognition could be used to interact with mobile devices using a sort of ‘gesture language’ based on multiple hand poses and gestures such as an open hand, swipe or thumbs up.

In the follow-up project, the amount of gestures were reduced (and a new gesture was proposed) after realizing that the gesture language was too complicated for people to grasp. After all, the technology was still not widely available and most people had no experience interacting using hand gestures in an AR or VR context.

But even after simplifying the gesture language, people still had difficulties understanding and using our software in an augmented reality context. As an attempt to solve this, I decided to design and develop a simple tutorial app, to help people learn the gestures and perform basic interactions using our software.

Understanding the problem

My hypothesis was that gesture interaction for AR was still too novel of an interaction and that people have difficulties “translating” their current ways of interacting with different devices (touching screens or buttons) into a new medium.

Learning step-by-step

Based on previous experience from user tests, I decided not to present an application with real content, but rather a sandboxy environment where people would first learn the gestures and interactions before applying their newly learned skills in an AR application with real content.

I wanted to make sure people would fully understand each step before moving on to the next one so I broke the tutorial down into several steps for the user to learn:

Summoning and moving the cursor

The base for initiating and performing all other interactions.

Learning the camera boundaries

If the device camera can't see the hand, the cursor can not be used.

Select by tapping/clicking

On buttons or other UI elements.


For browsing content such photos in a gallery, lists or web pages.

Testing & iteration

I tested my tutorial application on 12 people before letting them test one of our applications with real content and all 12 could navigate and interact with the content with issues, a huge improvement compared to previous user studies and A/B tests I had performed with our applications.