Magic Pen: automatic pen-mode switching for document annotation
Desousa, Kevin A.
MetadataShow full item record
Traditional digital pen interfaces use menu buttons to change pen modes, resulting in time and cognitive load spent on round-trips and potential errors from tapping small mode selection buttons. This thesis presents Magic Pen, a technique that automatically switches between digital pen modes. The Magic Pen system is driven by a Long Short-Term Memory (LSTM) model trained on pen data collected from nine participants and uses Transfer Learning (TL) to tune itself towards a user’s specific annotations iteratively. If Magic Pen chooses the incorrect mode, mitigation techniques incorporate flick gestures and screen taps to correct or remove a stroke. An annotation environment was also developed to rapidly prototype annotation systems with user-supplied documents. Magic Pen was originally evaluated with 18 participants and further evaluated with 8 participants. Magic Pen was preferred over a more conventional menu approach, and using TL allowed for greater model predictability and stability.