GESTURE RECOGNITION BASED ON DEEP LEARNING: APPLICATION TO 3D INTERACTION IN COLLABORATIVE MIXED REALITY

Abstract

In virtual reality (VR) and augmented reality (AR), the word interaction is used to designate a set of rules and techniques that allow the user to perform interaction tasks in a virtual environment. The use of hand gestures is the most natural way to perform an interaction, for this reason gestures recognition has a part in the world of interaction. In this context, our subject aims to develop a 3D interaction collaborative application by hand gestures in virtual or augmented reality, using gesture recognition by neural network. To achieve our work, we have used many software tools such as visual studio and Unity 3D for the environment and Vuforia SDK for augmented reality, So we used these tools to design a virtual environment and represent virtual hands that reproduce user’s hands gestures. In the purpose to allow the user to interact in a virtual or augmented environment we have implemented a 3D interaction techniques which is “simple virtual hand" for the three main tasks of interaction: navigation, selection and manipulation. Also we have implemented a network manager to allow users to collaborate with each other. This memory contains a state of the art that expose the necessary concepts for understanding our work, and a detailed explanation of the implementation leading to results and a presentation of all the tools used.