Such a cool and inspirational project! Regarding the drift on pinch, have you tried storing the pointer position of the last second and use that as the click position? You could show this position as a second cursor maybe? I've always wondered why Apple doesn't do this for their "eye moves faster than hands" issue as well.