Choice or circumstance? Adjusting measures of foreign policy similarity for chance agreement

The similarity of states' foreign policy positions is a standard variable in the dyadic analysis of international relations. Recent studies routinely rely on Signorino and Ritter's (1999, Tau-b or not tau-b: Measuring the similarity of foreign policy positions. International Studies Quarterly 43:115-44) S to assess the similarity of foreign policy ties. However, S neglects two fundamental characteristics of the international state system: foreign policy ties are relatively rare and individual states differ in their innate propensity to form such ties. I propose two chance-corrected agreement indices, Scott's (1955, Reliability of content analysis: The case of nominal scale coding. The Public Opinion Quarterly 19:321-5) π and Cohen's (1960, A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37-46) κ, as viable alternatives. Both indices adjust the dyadic similarity score for a large number of common absent ties. Cohen's j also takes into account differences in individual dyad members' total number of ties. The resulting similarity scores have stronger face validity than S. A comparison of their empirical distributions and a replication of Gartzke's (2007, The capitalist peace. American Journal of Political Science 51:166-91) study of the 'Capitalist Peace' indicate that the different types of measures are not substitutable.