Order, order! How sorting order of online discussions affects deliberative quality and opinion expression

Researchers have been concerned about the deliberative quality of debate within discussion forums on websites and social media. News organizations, social media and discussion platforms use a wide variety of possibilities for sorting posts in debate forums. This paper studies whether the order in which comments are sorted affect the deliberative quality of subsequent debates. Research has shown that different ways of sorting contributions can direct collective attention to very different types of information. Due to social influence bias like herding effects and position bias, posts on top of a ranking will receive more attention than posts lower down the lists (Hogg and Lerman, 2014; Lerman and Hogg, 2014). It is therefore expected that sorting order impacts the behavior of the participants in the debate. It is expected that when posts of a lower deliberative quality are place on the top of the discussion this will reinforce uncivil communication styles and polarization. In the fall of 2020, a pre-registered experiment was conducted in Denmark to see how different ways of sorting posts in online discussions on sexism and the legalization of euthanasia impact the behavior and perceptions of participants (https://aspredicted.org/blind.php?x=2vj9s2). A representative sample of the Danish population consisting of 1200 participants got to see 60 comments, where either comments with low deliberative quality were placed on top or comments in which participants speak out and state a clear opinion were shown higher up. The participants were then are asked to (1) ‘like’ the comments which they approve of; (2) write a comment themselves; and (3) assess the deliberative quality and feelings associated to the debate. The comments which the participants wrote were later coded for deliberative quality and outspokenness, based on a tested codebook with satisfactory intercoder reliability scores (Krippendorf’s Alpha). The results showed that sorting order impacts the behavior of the participants in the discussion. In each condition, comments high up in the forum receive more likes than comments lower in the discussion, independent of the deliberative quality of these comments. Supporting the expectations, comments with low deliberative quality received more likes when they were placed high in the forum compared to when they were spread randomly through the discussion. This was found for both topics (sexism and the legalization of euthanasia). Similarly, when comments were ranked based on outspokenness, comments taking a stand received more likes. For the discussion on the legalization of euthanasia, the sorting order of the comments also affected the perceived deliberative quality of the debate and lead to either feelings of irritation or optimism. For the discussion on sexism this was not the case. Sorting order did not affect the deliberative quality or outspokenness of the comments which the participants themselves wrote. In sum, the experiments show that sorting order can affect deliberative quality of online discussions. The implications for discussion moderators, news organizations and social media platforms are discussed.