What does expected agreement represent in reliability analysis?

• A. The observed proportion of identical results across all pairs.
• B. The agreement expected by chance for positive results only.
• C. The agreement expected by chance for negative results only.
• D. The agreement expected by chance for both positives and negatives combined.

Answer: (D)

Expected agreement is the portion of agreement you would expect to see purely by chance, given how often each rater tends to label cases as positive or negative. It accounts for both outcomes, not just one category. To calculate it, multiply the probability that both raters label a case as positive and add the probability that both label it as negative. For example, if one rater labels positives 60% of the time and negatives 40%, while the other rater labels positives 50% and negatives 50%, the chance agreement would be 0.6×0.5 + 0.4×0.5 = 0.30 + 0.20 = 0.50. This expected agreement serves as a baseline to interpret observed agreement and is used in measures like kappa to assess agreement beyond what would be expected by chance.