New Models Based on BERT for Toxicity Detection

Abstract

This present study is dedicated to contribute on textual toxicity detection task which is interested on detecting texts including any kind of intentional abusiveness. In this study, we propose an approach based on deep learning and pre-trained BERT model for enhancing automated toxicity detection. In our work, we test several BERT based models built by either adding DL layers or by composing EL models constituted of BERT model and other DL models trained individually. We also investigate which strategy is better for dealing with class imbalance problem faced on datasets treating toxicity between data resampling or using focal loss for training. The best results reached on English dataset is of 98,6124 for macro average ROC-AUC score, where the best proposed model detected more than 87% of toxic texts. On the Arabic data, the best precision, recall, and f1-measure scores achieved are respectively 76%, 66%, and 71%.