Module 4: Study Design Measures
How To Create a Research Methodology
When formulating methodology, it is critical to consider the types of methods that will answer the research questions. For example, if one wishes to assess the efficacy of an educational intervention the “change in knowledge” must be measured. This lends itself to a pre-test/post-test methodology in which the research will determine the knowledge of study participants on a topic prior to the intervention and then again after the educational intervention has been implemented. However, in order to determine the actual impact of an intervention, a pre-test/post-test methodology must always be compared with a control group. For more information about this, please see the section on Assessing Behavioral Changes: The Importance of Having a Baseline For Comparison.
Measures are the items in a research study to which the participant responds. They can be survey questions, interview questions, or constructed situations, to name a few. When constructing interviews and surveys, it is important that the questions directly relate to the research questions. Furthermore, it is important that the surveys and interviews are not extremely time-consuming (ideally within a 20-30 minute limit). If an interpreter will be used, simple questions are always better and easily interpreted questions that avoid ambiguity will lead to more accurate results. Lastly, instead of creating a survey it is better to do research to find out if a similar study has been conducted. If so, previous surveys should be used to yield standardized measures for comparison. Irrespective of the form that these measures take, there are several important design elements that are required to make the study effective.
Study measures should:
- Take into account the characteristics of the participant.
- Use informal language that someone with no experience in the field can understand.
- Be respectful of the cultural context in which the participant has shaped his or her worldview.
- Provide neither too much nor too little information. Too much information can be an unnecessary distraction, while too little information leads to ambiguity and potential misinterpretation of the study measure.
- Be brief and specific.
- Avoid negations, as they can lead to mistakes and can be difficult to understand.
- Avoid double-barreled questions (questions that ask two questions in one, such as “Do you support the government’s decision to cut spending to police training and after school programs?”). If a participant would answer “yes” to one part but “no” to another, requesting an answer of “yes or no” to the pair as a whole invalidates the measure.
- Use multiple questions to assess the same construct. For example, simply asking a person if they feel “good” about themselves as a measure of self-esteem is not enough. Instead, asking that person several questions about body image, self-worth, and self-evaluation can help paint a better picture of how they are really feeling.(1),(2),(3)
Self-report measures, whether in survey form or in interview form, are easily affected by several biases that the participant may exhibit, and for this reason must be designed carefully by the researcher. The researcher also must be cautious about what kinds of conclusions are drawn from self-report measures. Some potential issues with self-report are:
Social desirability bias: Participants are usually uncomfortable or unwilling to share information that does not reflect well on them in their social environment, even if they know their responses are entirely anonymous. For example, participants may understate or overstate the extent to which they experience a certain feeling, depending on how socially appropriate or desirable that feeling. Researchers must do their best to make it abundantly clear that anonymity will be preserved for the participant, and honesty must be encouraged. Researchers should also lead with less-intimidating questions to make the participant feel more comfortable before asking anything that might be more difficult to answer honestly. Another option is to structure the question in such a way as to normalize the behavior: “As you know, many people do X… To what extent do you do X?”
Self-evaluation biases: Participants will sometimes bend their answers on self-report measures to better reflect how they “think they should be” rather than how they actually are. This is similar to the social desirability bias, but is more difficult to overcome because anonymity is not the issue here. Instead, bias results from the participant’s evaluation of him or herself. The researcher’s best course of action is to encourage honesty and normalize the behavior or feeling as reviewed above.
Forgetfulness: Sometimes researchers ask participants about their past experiences or feelings without considering the fact that human memory is very plastic. People’s recollections may be inaccurate, and it is important for a researcher to consider this when designing study measures. Self-report measures do play an important role in research, although they should be used with caution. They are essential in situations where the researcher is asking about a participant’s self-concept or seeking to study the specificities of a participant’s experience. Self-report is also very useful for logistical reasons, as it is often the simplest of methods to implement and requires the least resources.(4)(5)
(1) Eibach, R. “Scale Questions: Simple Questions, Complex Answers” Lecture at Yale University 10/2/08
(2) Pelham, B. W.; Blanton, H. Conducting Research in Psychology: Measuring the Weight of Smoke, 3rd Edition. Wadsworth Publishing (February 27, 2006)
(3) Trochim, W. M. K. “Survey Research” Research Methods Knowledge Base 2nd Edition. Accessed 2/24/09 http://www.socialresearchmethods.net/kb/survey.php
(4) Pelham, B. W.; Blanton, H. Conducting Research in Psychology: Measuring the Weight of Smoke, 3rd Edition. Wadsworth Publishing (February 27, 2006)
(5) Eibach, R. “Scale Questions: Simple Questions, Complex Answers” Lecture at Yale University 10/2/08