Internal validity - how valid were the procedures that I followed in terms of the research method to obtain the information I need External validity - how generalizable is the information learned from a study

Internal validity

• History effect - did something recently happen to one of the sample groups that would make it differentiate from the other groups. For example, if you ask small children about their fears of ghosts, some of the children may report less fear if they are surveyed near Halloween.
• Maturation - when the study doesn't control for natural growth; in other words, the natural maturation of the group is the actual cause of improvement in the study instead of the treatment
• Testing - what are the effects of a pretest on a posttest; familiarity with the instrument impacts the results on the second test
• Instrumentation - is an issue when instrumentation changes during a study; for example, calibration of individual opinions in a observation study could have differences in results just because a one rater might think an event is a 5 on a 1-10 scale, and another rater might think the same event is a 7. The result grades on essay papers depending on the mood of the instructor. If the publisher comes out with a new version of a test and the previous test isn't available. Then you need to see how closely the two tests are correlated.
• Statistical regression - also known as regression to the mean; for example, getting a perfect score on the SAT only leaves the possibility of going down in score. The more extreme the results, the more likely each is to move towards the mean because of the error that put them in the extreme in the first place like guessing on questions that make the difference between a perfect score and a lower score.
• Differential selection - people that you pick are not representative of the population in the first place;
• Experimental mortality - differentially loose respondents; for example, you might not even know who you lost or picked in the first place. Loosing the extremes in both cases of a paired study could be the actual cause of change rather than the treatment
• Selection maturation interaction - as part of the selection, the sample actually has unknown differences that make the differences in the results instead of the treatment. For example, if teaching children basketball, while the children may start with similar free-throw ability, one child may improve faster than another simply because of superior hand-eye coordination, rather than the basketball instruction.

External validity

• Interactive or reactive effects of testing - example: giving a pretest to the people in the population sample may impact their performance during observation, so the pretest has an interaction effect that limits the generalizability of the results to the general population because the whole population did not take the pretest.
• Interaction and selection biases - the treatment only works for the selection, so can't be generalized; may be a result of the environment the selected group was performing in
• Reactive effects of experimental arrangements
• Multiple treatment interference - when you have more than one treatment, how do you know which one made the difference? Something like sequencing presentation of treatments may make a difference.

Bibliography

Campbell, D. T. & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago, R. McNally.