Direct speech detection using deep learning

About the project

The project was born as part of Deep learning in computational stylistics collaboration between University of Antwerp and Institute of Polish Language PAS, funded by Research Foundation of Flanders (FWO) and the Polish Academy of Sciences (PAS), and in support of the COST Action Distant Reading in which some of us participate.

The outline

Fictional prose can be broadly divided into narrative and discursive forms with direct speech being central to any discourse representation (alongside indirect reported speech and free indirect discourse). This distinction is crucial in digital literary studies and enables interesting forms of narratological or stylistic analysis. The difficulty of automatically detecting direct speech, however, is currently under-estimated. Rule-based systems that work reasonably well for modern languages struggle with (the lack of) typographical conventions in 19th-century literature. While machine learning approaches to sequence modeling can be applied to solve the task, they typically face a severed skewness in the availability of training material, especially for lesser resourced languages.
In our project, we develop a multilingual approach to direct speech detection in a diverse corpus of 19th-century fiction in 9 European languages. The proposed method finetunes a transformer architecture with multilingual sentence embedder on a minimal amount of annotated training in each language, and improves performance across languages with ambiguous direct speech marking, in comparison to a carefully constructed regular expression baseline.

Malaga workshop materials

The presentation
Materials on binder
The latest model - as of late February

Resources

Byszuk, J., Woźniak, M., Kestemont, M., Leśniak, A., Łukasik, W., Šeļa, A. and Eder, M. (2020). Detecting direct speech in multilingual collection of 19th century novels. Proceedings of the LREC 2020. Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2020). Marseille, pp. 100–04.