Which statement best describes the concept of expected agreement in reliability studies?

• A. It is equal to the observed agreement.
• B. It is always higher than the observed agreement.
• C. It estimates agreement by chance for positive results only.
• D. It estimates agreement by chance for both positive and negative results.

Answer: (D)

Expected agreement represents the portion of observed concordance that would occur just by chance, given how often each rater labels items as positive or negative. In reliability studies with binary classifications, two raters will agree sometimes simply because they both lean toward calling things positive or both lean toward calling things negative. This chance agreement is computed from the marginal proportions of each category for both raters, and it includes agreements in both positive and negative categories, not just one. This is the baseline used to adjust the observed agreement when estimating how much agreement is beyond what luck would produce, as in a kappa statistic.

So, the statement that best describes expected agreement is that it estimates agreement by chance for both positive and negative results. It isn’t equal to observed agreement, isn’t guaranteed to be higher than observed, and it doesn’t consider only positive results.