Validating Sentiment Analysis with Calls to Order in Parliament

The paper describes a supervised machine learning approach for sentiment analysis based on distributed word embeddings to measure the level of (in)civility in parliamentary speeches. Going beyond the dominant bag-of-words approach for modeling these features, we test sentiment analysis based on distributed representations of words with a neural network classification model. To validate this automated measurement, we present a substantive application. Using parliamentary speeches from the last two decades we attempt to implement a hard task, namely to predict calls to order in plenary debates in the Austrian national parliament. Calls to order are mostly keyword based censures by the presiding officer. As there are three different presiding officers elected from different parties we will also attempt to evaluate their fairness in dealing with incivility in debates.