Piano AR: A Markerless Augmented Reality Based Piano Teaching System

This paper presents a novel markerless augmented reality based piano teaching system, which tracks the real keyboard of piano naturally. Following the virtual hands augmented on the keyboard, beginners of piano can practice playing piano. Instead of the marker in traditional AR system, the geometrical parameters of the piano keyboard are detected to calculate the transformation matrix from keyboard coordinate to camera coordinate. We first extract all the contours of the image, from which possible contours of keyboard are found. Then the area of keyboard is identified by the structure of white and black keys predefined. We evaluated the system with speed and accuracy. As a result, the system operates at 15fps, and the average error worked out by our method is about 1.97 pixels. The usefulness of our approach is reflected in an example that a user learns to play piano.