We present a novel approach for extending the input space around unmodified mobile devices. Using built-in front-facing cameras of unmodified handheld devices, GlassHands estimates hand poses and gestures through reflections in sunglasses, ski goggles or visors. Thereby, GlassHands creates an enlarged input space, rivaling input reach on large touch displays. We introduce the idea along with its technical concept and implementation. We demonstrate the feasibility and potential of our proposed approach in several application scenarios, such as map browsing or drawing using a set of interaction techniques previously possible only with modified mobile devices or on large touch displays. Our research is backed up with a user study. In consecutive research we discuss, how to extend this beyond the use of reflective glasses.


30 Second Teaser

Main Video (4:22 min)


  • Jens Grubert, Eyal Ofek, Michel Pahud, Matthias Kranz and Dieter Schmalstieg. GlassHands: Interaction Around Unmodified Mobile Devices Using Sunglasses. In Proceedings of the 2016 ACM Conference on Interactive Surfaces and Spaces (pp. 215-224). preprintdoi
  • Jens Grubert, Eyal Ofek, Michel Pahud, Matthias Kranz and Dieter Schmalstieg. Towards Interaction Around Unmodified Camera-equipped Mobile Devices. In Cross-Surface Workshop in Conjunction with the 2016 ACMConference on Interactive Surfaces and Spaces. preprint | workshop website