Automating Identification of Toxic Comments in Student Evaluations
Student evaluations of teaching (SET) play a crucial role in assessing instructor effectiveness by providing qualitative feedback through student comments. However, these comments primarily reflect extreme opinions, with toxic comments being particularly memorable. Toxic comments include unhelpful, hurtful, harassing, or personally attacking remarks that can negatively impact instructors and influence hiring/firing decisions unfairly.
Most universities currently lack a systematic approach to identify and manage toxic comments in SETs. This raises the question of how effectively automated machine learning methods can classify toxic comments. To address this, a 20-item codebook was developed to train human coding for labeling SET comments as toxic across three universities. The data from this coding process were used to evaluate various toxicity classifiers.
Human Coding and Machine Learning Models
The human coding process demonstrated moderate to strong intercoder reliability, a crucial factor for data classification. Among the machine learning models evaluated, the pre-trained Perspective model stood out for its ability to minimize false positives and maximize true positives, surpassing traditional classifiers. This suggests that machine learning can efficiently identify and eliminate toxic comments in student evaluations.
It is essential to ensure the reliability and accuracy of toxicity classification given the impact these comments can have on instructors and academic institutions. The use of advanced machine learning models can enhance this process and contribute to a more constructive feedback environment.
For further details on this study, please log in via an institution to access the full article.
Additionally, the complete data and scripts used in this study will be available on a GitHub repository for replication purposes.
