Measuring the Depth of Trade Agreements with Large Language Models

Depth, as the extent to which an international agreement restrains states’ behavior, has long been recognized as a key dimension of international institutions. This concept is crucial for evaluating institutional effectiveness and compliance, as shallow agreements (low depth) may yield high compliance rates without substantive behavior change. Depth is also central to explaining why governments even sign up to international institutions. In this study, we leverage the Design of Trade Agreements (DESTA) dataset of preferential trade agreements (PTAs) to explore how well large language models (LLMs) can measure the depth of international institutions. The DESTA data provide widely used, expert-coded indices of PTA depth, making it an ideal benchmark for testing automated text analysis. We apply a 20- billion-parameter open-source LLM (GPT-OSS-20B) to the full texts of hundreds of PTAs, generating several new measures of agreement depth. We then compare these LLM-generated depth scores to each other and to the original DESTA depth indices using scatterplot visualizations and correlation analysis. Results show that the automated LLM measures closely replicate the established depth rankings of agreements, indicating strong convergent validity. However, we also find some nuanced deviations that suggest the LLM measures may be capturing subtle aspects of “depth” beyond the scope of the manual coding. Beyond substantive findings on trade agreements, our work offers broader methodological insights by demonstrating that modern LLMs can function as effective coders of complex legal texts. This approach opens new avenues for political scientists: by harnessing LLMs to systematically analyze international treaties and other legal documents, researchers can dramatically extend the scale and scope of institutional analysis while maintaining accuracy and detail.