The “Cohen” bit comes from its inventor, Jacob Cohen. Kappa (κ) is the Greek letter he used to designate his measure (others used Roman letters, z.B. the “t” in “t-test”, but the measure of conformity conventionally use Greek letters). The R command is kappa2 and not kappa, because the kappa command also exists and does something completely different that, by chance, uses the same letter to represent it. It probably would have been better to call the order something like cohen.kappa, but they didn`t. The most important result here is %-agree, which is your percentage of match. The edition also tells you how many topics you`ve reviewed and how many people have given reviews. The bit, which says tolerance = 0, refers to an aspect of the percentage concordance that is not covered in this course. If you are interested in the tolerance of a percentage agreement calculation, type ?agree in the console and read the help file for this command. We can now use the command to establish a match percentage. The agree command is part of the wrong package (abbreviated for Inter-Rater Reliability), so we must first load this package: So, on a scale from zero (chance) to one (perfect) your approval in this example was about 0.75 – not bad! 50% agree, it`s much more impressive when, say, there are six options. In this case, imagine that both evaluators roll dice.

Once in six, they would get the same number. So the percentage of match by chance, if there are six options, is 1/6 – about 17% approval. If two reviewers agree in 50% of the time, if they use six options, this match is much higher than we expected by chance. Multiply the quotient value by 100 to get the percentage of concordance for the equation. You can also move the decimal to the right two places, which gives the same value as multiplying to 100. For example, multiply 0.5 by 100 to get a percentage of 50%. If you have multiple reviewers, calculate the percentage agreement as follows: As you can probably see, calculating percentage chords can quickly prove to be a good one for more than a handful of reviewers. For example, if you had 6 judges, you would have 16 pair combinations to calculate for each participant (use our combination calculator to find out how many couples you would get for multiple judges).

For example, if you want to calculate the percentage of concordance between the numbers five and three, take five minus three to get the value of two for the counter. Cohens Kappa is a measure of concordance, which is calculated as in the example above. The difference between Cohen`s kappa and what we just did is that Cohen`s Kappa also deals with situations where evaluators use some of the categories more than others. This affects the calculation, as it is likely that they agree by chance. For more information on this, see Cohens Kappa. There are a few words that psychologists sometimes use to describe the degree of correspondence between evaluators, depending on the Kappa value they get. These words are: Jacob Cohen thought it would be much more suitable if we could have a degree of agreement where zero always meant the degree of agreement expected at random and 1 always meant a perfect match. This can be achieved by the following sum: the basic measure of inter-board reliability is a percentage of agreement between evaluators. “What is the reliability of the InterRater?” is a technical way of asking, “How much do people agree?” . . . .