Identifying student difficulty and frustration from discussion forum postings

Abstract

This work applies natural language processing techniques, like those used in sentiment analysis, to the data generated by students in a digital online learning environment to detect confused or frustrated students and alert instructors so that time-sensitive educational support can be provided. Utilizing a data set of 9,141 discussion posts collected from an Introduction to Java Programming course, seven types of classifiers were tested, including Support Vector Machine (SVM), Naive Bayes, and Random Forest algorithms; it was determined that the optimum results for the data set was an SVM classifier using a non-linear Gaussian kernel, combined with a custom dictionary and noun phrase POS frequency count for feature vector identification and the determination of a relevance probability. The resulting application, TutorAlert, produced a promising F1 score of 0.79 and an accuracy of 0.83. Further, agreement values of 88% were achieved during inter-rater reliability testing between the classifier and human judges.