Early predicting of student struggles using body language

The accuracy of RGB-D sensing has enabled many technical achievements in applications such as gamification, task recognition, as well as pedagogical applications. The ability of these sensors to track many body parts simultaneously has introduced a new data modality for analysis. By analyzing body language, this work can predict if a student will struggle in the future, and if an instructor should intervene. To accomplish this, a study is performed to determine how early (after how many seconds) does it become possible to determine if a student will struggle. A simple neural network is proposed which is used to jointly classify body language and predict task performance. By modeling the input as both instances and sequences, a peak F Score of 0.459 was obtained, after observing a student for just two seconds. Finally, an unsupervised method yielded a model which could determine if a student would struggle after just 1 second with 59.9% accuracy.