Module 3: Ensuring Validity
A confounding variable is an extraneous variable that is statistically related to (or correlated with) the independent variable. This means that as the independent variable changes, the confounding variable changes along with it. Failing to take a confounding variable into account can lead to a false conclusion that the dependent variables are in a causal relationship with the independent variable. Take, for example, a study that seeks to investigate the relationship between income levels and test scores. Without controlling for other variables, the study finds that higher income correlates with better test scores and concludes that the two must be directly related. This is a flawed conclusion because there are many lurking confounding variables that may influence this supposedly clear-cut relationship. For example, perhaps individuals at one school received better education than those at another school. Without controlling for the confounding variables of education level and quality of education, the relationship between income level and test scores cannot be assumed.
Before we begin discussing the specifics of data analysis, let’s pause for a moment to discuss the validity of a study and what it means. For the rest of this short course in research methods, we will be stopping to enumerate the various threats to validity that exist at each stage of the research process. We have already seen one: failure to properly operationalize the variables of interest can lead to the researcher drawing inappropriate conclusions about the research question. If, for example, the researcher had chosen to operationalize “economically productive” as “amount of money a person has in his or her savings,” the researcher would have observed an entirely different result. It is possible for people to have other sources of income (gifts, spouses, inheritances, etc.) that may affect this variable, meaning that it is not a good measure of what is intended to be measure and therefore is not a good example of operationalization. But what exactly is validity?
Generally speaking, validity refers to whether or not a study is well designed and provides results that are appropriate to generalize to the population of interest. There is a lot more to validity that we will further discuss in this course, and Trochim’s “Research Methods Knowledge Base” provides a succinct and useful summary of each of the kinds of validity.(1) There are three types of validity with which a researcher should be concerned.
Internal validity applies in studies that seek to establish a causal relationship between two variables, and it refers to the degree to which a study can make good inferences about this causal relationship. The essence of internal validity is whether or not a researcher can definitively state that the effects observed in the study were in fact due to the manipulation of the independent variable and not due to another factor. “Third variables” that the researcher may not consider or may not be able to control can affect the outcome of a study and can therefore prevent internal validity. A study is considered to be internally valid if the researcher can demonstrate that variable caused the observed effect.(2)
Construct validity is closely related to the process of operationalizing which we discussed in Module 1. It refers to the extent to which a researcher can claim that accurate inferences can be made from the operationalized measures in a study for the theoretical constructs on which they were based. Construct validity is concerned with generalizing from the specificities of a study to the broader concept that the study attempts to measure or draws conclusions. A study is considered to have construct validity if the researcher can demonstrate that the variables of interest were properly operationalized.(3)
A researcher often cannot work with the entire population of interest, but instead must study a smaller sample of that population in order to draw conclusions about the larger group from which the sample is drawn. External validity is concerned with the extent to which the conclusions can be generalized to the broader population. A study is considered to be externally valid if the researcher’s conclusions can in fact be accurately generalized to the population at large.(4)
As a researcher, it is important to keep the concept of validity in mind at all times when designing a study. A good researcher will discuss the project design with an advisor or a group of colleagues to help ensure that validity is preserved at every stage of the process. A research project that lacks validity may draw conclusions that are inappropriate or even dangerous if applied to the target population.
For more information about how to ensure the validity of research, please review Research Validity.
(1) Trochim, W. M. K. “Design” Research Methods Knowledge Base 2nd Edition. Accessed 2/24/09 http://www.socialresearchmethods.net/kb/design.php
(4) Pelham, B. W.; Blanton, H. Conducting Research in Psychology: Measuring the Weight of Smoke, 3rd Edition. Wadsworth Publishing (February 27, 2006)