Writing up reliability results

Guiding Principle: No measure is ever reliable. Scores determined by a measure may be reliable depending on who completes the measure and the conditions under which the measure is completed (when and where).

Structure of the Reliability write-up

Procedure One Procedure Two

1. Comment on the reliability of the scores by examining the reliability coefficient.

Everyone has her/his own standards, but here are what mine tend to be:

< .65 Poor;
.65 ≤ Modest ≤ 80;
> .80 Very good

 1. Comment on reliability coefficient

2. Comment on the corrected item-total correlations
  1. Note which items have a negative corrected item total correlation.
  2. Note whether any negative corrected item total correlation are attributable to miscoding or items with response options are scaled opposite of the majority of items.
  3. Note which items have a zero corrected item total correlation.
  4. Note which items with a zero corrected item total correlation do so because of no variance (every person responded the same way to the item)
2. Do nothing more 3. Comment on how high your reliability coefficient would increase if you were to drop out all negative item-total correlations one at a time until you had no more negative corrected item-total correlations.
  4. Do nothing more

Sample paragraph

Responses for the various features of the agency website were judged to be moderately reliable for the customer service representatives who participated in the survey, with a reliability coefficient of 0.778. A review of the corrected item-total correlations suggests that the questions for the website as a critical tool and a unique website in the marketplace do not correlate with the corrected total very well. Their elimination is warranted on the basis that reducing the scale to only relevant items would make for a better, more parsimonious scale. It turns out that removing the item may further be motivated by anticipated increase in the reliability coefficient reported in the output (0.884).

If HTML newsletters, online payments, Internet as a tool, and customer communication with the agency were removed, the reliability of the scale would increase to 0.963.

To examine the impact of removing both items, each item was removed one at a time. This approach is necessary because the impact of removing one item changes the relationship of the other item with the changing total.