Tutorial - Perspectivist NLP: Data, Models, Evaluation, and LLMs

This tutorial is designed to provide the basic tools to start working within the perspectivist paradigm in NLP. The first part will illustrate the fundamental differences between traditional data sets and disaggregated data collections suited for use in perspectivist methods. We will cover standard disagreement measures, as well as recent measures of structured and systematic disagreement. The tutorial continues with a hands-on experiment on supervised perspectivist classification.
The focus will shift towards the issues around evaluation of perspectivist methods, introducing a recent theoretical framework and its implementation, with a hands-on exercise. Finally, We will cover how eprspectivist NLP relates to modern LLMs in a generative, inference-only setting.