In search of solid ground – A novel validation framework for text-based measures of social science constructs

Nowadays, text-based methods are widely used to detect and measure social science constructs from textual data, such as emotions, issues or political attitudes. However, the validity of text-based measures can be difficult to assess, given the complex and often unstructured nature of the data involved. This often leads to confusion about how to validate text-based measures, ultimately leading to fragmented and inconsistent validation strategies. Building on recent work from a systematic review of validation practices and qualitative expert interviews, this paper presents a novel validation framework to guide researchers who want to measure latent constructs using textual data. The framework draws on the three stages of validation practice derived from the psychological testing literature (substantive, structural and external phase), but extends them to the domain of text analysis. At the beginning of the research process, the substantive phase encompasses the theoretical underpinnings of a measure and includes all considerations of the appropriateness of the research process in light of its theoretical underpinnings. The structural phase entails qualitative and quantitative validation steps to examine the semantic properties of the respective text model and its measures. Finally, the external phase includes any evidence of how the construct relates to other measures or criterion data. To illustrate the applicability of the framework, the paper also presents a case study that demonstrates the suitability and flexibility of the proposed framework. Overall, this paper aims to serve as a valuable resource for researchers in the field of political science research and to contribute to the ongoing efforts to improve the quality and reliability of text-based measures of social science constructs.