Essentials of Research Design and Methodology Geoffrey Marczyk David DeMatteo David Festinger

of Research Design and Methodology Geoffrey Marczyk David DeMatteo David Festinger Copyright © 2005 by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appro- priate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the web at . Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accu- racy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, includ- ing but not limited to special, incidental, consequential, or other damages. This publication is designed to provide accurate and authoritative information in regard to the sub- ject matter covered. It is sold with the understanding that the publisher is not engaged in rendering professional services. If legal, accounting, medical, psychological or any other expert assistance is required, the services of a competent professional person should be sought. Designations used by companies to distinguish their products are often claimed as trademarks. In all instances where John Wiley & Sons, Inc. is aware of a claim, the product names appear in initial cap- ital or all capital letters. Readers, however, should contact the appropriate companies for more com- plete information regarding trademarks and registration. For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. For more information about Wiley products, visit our web- site at . Library of Congress Cataloging-in-Publication Data: Marczyk, Geoffrey R., 1964– Essentials of research design and methodology/Geoffrey Marczyk, David DeMatteo, David Festinger. p. cm.—( Essentials of behavioral science series) Includes bibliographical references and index. ISBN 0-471-47053-8 ( pbk.) 1. Psychology— Research— Methodology. I. DeMatteo, David, 1972– II. Festinger, David. III. Title. IV. Series. BF76.5.M317 2005 150.72— dc22 4 Printed in the United States of America. 10 9 8 7 6 5 4 3 2 1 To Helene and my family G.M. To Christina and Emma D.D. To Tracy, Ashley, and Elijah D.F. Series Preface ix Acknowledgments xi One Introduction and Overview 1 Two Planning and Designing a Research Study 26 Three General Approaches for Controlling Artifact and Bias 65 Four Data Collection, Assessment Methods, and Measurement Strategies 95 Five General Types of Research Designs and Approaches 123 Six Validity 158 Seven Data Preparation, Analyses, and Interpretation 198 Eight Ethical Considerations in Research 233 Nine Disseminating Research Results and Distilling Principles of Research Design and Methodology 261 References 277 Index 283 vii n the Essentials of Behavioral Science series, our goal is to provide readers with books that will deliver key practical information in an efficient, ac- cessible style. The series features books on a variety of topics, such as statistics, psychological testing, and research design and methodology, to name just a few. For the experienced professional, books in the series offer a concise yet thorough review of a specific area of expertise, including nu- merous tips for best practices. Students can turn to series books for a clear and concise overview of the important topics in which they must become proficient to practice skillfully, efficiently, and ethically in their chosen fields. Wherever feasible, visual cues highlighting key points are utilized alongside systematic, step-by-step guidelines. Chapters are focused and succinct. Topics are organized for an easy understanding of the essential material related to a particular topic. Theory and research are continually woven into the fabric of each book, but always to enhance the practical application of the material, rather than to sidetrack or overwhelm readers. With this series, we aim to challenge and assist readers in the behavioral sciences to aspire to the highest level of competency by arming them with the tools they need for knowledgeable, informed practice. The purposes of Essentials of Research Design and Methodology are to dis- cuss the various types of research designs that are commonly used, the ba- sic process by which research studies are conducted, the research-related considerations of which researchers should be aware, the manner in which the results of research can be interpreted and disseminated, and the typi- ix x SERIES PREFACE cal pitfalls faced by researchers when designing and conducting a research study. This book is ideal for those readers with minimal knowledge of re- search as well as for those readers with intermediate knowledge who need a quick refresher regarding particular aspects of research design and methodology. For those readers with an advanced knowledge of research design and methodology, this book can be used as a concise summary of basic research techniques and principles, or as an adjunct to a more ad- vanced research methodology and design textbook. Finally, even for those readers who do not conduct research, this book will become a valuable addition to your bookcase because it will assist you in becoming a more educated consumer of research. Being able to evaluate the appropriate- ness of a research design or the conclusions drawn from a particular re- search study will become increasingly more important as research be- comes more accessible to nonscientists. In that regard, this book will improve your ability to efficiently and effectively digest and understand the results of a research study. Alan S. Kaufman, PhD, and Nadeen L. Kaufman, EdD, Founding Editors Yale University School of Medicine We would like to thank Karen Dugosh and Audrey Cleary for their help- ful comments on earlier drafts of this book. We would also like to thank Susan Matties for her research assistance. Additional thanks go to Dr. Vir- ginia Brabender for introducing us to John Wiley and Sons. Finally we’d like to thank Tracey Belmont, our editor, for her support and sense of humor. xi Essentials of Research Design and Methodology One rogress in almost every field of science depends on the contribu- tions made by systematic research; thus research is often viewed as the cornerstone of scientific progress. Broadly defined, the purpose of research is to answer questions and acquire new knowledge. Research is the primary tool used in virtually all areas of science to expand the fron- tiers of knowledge. For example, research is used in such diverse scientific fields as psychology, biology, medicine, physics, and botany, to name just a few of the areas in which research makes valuable contributions to what we know and how we think about things. Among other things, by con- ducting research, researchers attempt to reduce the complexity of prob- lems, discover the relationship between seemingly unrelated events, and ultimately improve the way we live. Although research studies are conducted in many diverse fields of sci- ence, the general goals and defining characteristics of research are typically the same across disciplines. For example, across all types of science, re- search is frequently used for describing a thing or event, discovering the relationship between phenomena, or making predictions about future events. In short, research can be used for the purposes of description, ex- planation, and prediction, all of which make important and valuable con- tributions to the expansion of what we know and how we live our lives. In addition to sharing similar broad goals, scientific research in virtually all fields of study shares certain defining characteristics, including testing hypotheses, careful observation and measurement, systematic evaluation of data, and drawing valid conclusions. 1 2 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY In recent years, the results of various research studies have taken center stage in the popular media. No longer is research the private domain of re- search professors and scientists wearing white lab coats. To the contrary, the results of research studies are frequently reported on the local evening news, CNN, the Internet, and various other media outlets that are acces- sible to both scientists and nonscientists alike. For example, in recent years, we have all become familiar with research regarding the effects of stress on our psychological well-being, the health benefits of a low- cholesterol diet, the effects of exercise in preventing certain forms of can- cer, which automobiles are safest to drive, and the deleterious effects of pollution on global warming. We may have even become familiar with re- search studies regarding the human genome, the Mars Land Rover, the use of stem cells, and genetic cloning. Not too long ago, it was unlikely that the results of such highly scientific research studies would have been shared with the general public to such a great extent. Despite the accessibility and prevalence of research in today’s society, many people share common misperceptions about exactly what research is, how research can be used, what research can tell us, and the limitations of research. For some people, the term “research” conjures up images of scientists in laboratories watching rats run through mazes or mixing chemicals in test tubes. For other people, the term “research” is associated with telemarketer surveys, or people approaching them at the local shop- ping mall to “just ask you a few questions about your shopping habits.” In actuality, these stereotypical examples of research are only a small part of what research comprises. It is therefore not surprising that many people are unfamiliar with the various types of research designs, the basics of how research is conducted, what research can be used for, and the limits of us- ing research to answer questions and acquire new knowledge. Rapid Ref- erence 1.1 discusses what we mean by “research” from a scientific per- spective. Before addressing these important issues, however, we should first briefly review what science is and how it goes about telling us what we know. INTRODUCTION AND OVERVIEW 3 Rapid Reference 1.1 What Exactly is Research? Research studies come in many different forms, and we will discuss sev- eral of these forms in more detail in Chapter 5. For now, however, we will focus on two of the most common types of research—correlational re- search and experimental research. Correlational research: In correlational research, the goal is to deter- mine whether two or more variables are related. (By the way, “variables” is a term with which you should be familiar. A variable is anything that can take on different values, such as weight, time, and height.) For example, a researcher may be interested in determining whether age is related to weight. In this example, a researcher may discover that age is indeed re- lated to weight because as age increases, weight also increases. If a corre- lation between two variables is strong enough, knowing about one vari- able allows a researcher to make a prediction about the other variable. There are several different types of correlations, which will be discussed in more detail in Chapter 5. It is important to point out, however, that a cor- relation—or relationship—between two things does not necessarily mean that one thing caused the other.To draw a cause-and-effect conclu- sion, researchers must use experimental research.This point will be em- phasized throughout this book. Experimental research: In its simplest form, experimental research in- volves comparing two groups on one outcome measure to test some hy- pothesis regarding causation. For example, if a researcher is interested in the effects of a new medication on headaches, the researcher would ran- domly divide a group of people with headaches into two groups. One of the groups, the experimental group, would receive the new medication be- ing tested.The other group, the control group, would receive a placebo medication (i.e., a medication containing a harmless substance, such as sugar, that has no physiological effects). Besides receiving the different medications, the groups would be treated exactly the same so that the re- search could isolate the effects of the medications. After receiving the medications, both groups would be compared to see whether people in the experimental group had fewer headaches than people in the control group. Assuming this study was properly designed (and properly designed studies will be discussed in detail in later chapters), if people in the experi- mental group had fewer headaches than people in the control group, the researcher could conclude that the new medication reduces headaches. 4 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY OVERVIEW OF SCIENCE AND THE SCIENTIFIC METHOD In simple terms, science can be defined as a methodological and systematic approach to the acquisition of new knowledge. This definition of science highlights some of the key differences between how scientists and non- scientists go about acquiring new knowledge. Specifically, rather than relying on mere casual observations and an informal approach to learn about the world, scientists attempt to gain new knowledge by making care- ful observations and using systematic, controlled, and methodical ap- proaches (Shaughnessy & Zechmeister, 1997). By doing so, scientists are able to draw valid and reliable conclusions about what they are studying. In addition, scientific knowledge is not based on the opinions, feelings, or intuition of the scientist. Instead, scientific knowledge is based on objec- tive data that were reliably obtained in the context of a carefully designed research study. In short, scientific knowledge is based on the accumulation of empirical evidence ( Kazdin, 2003a), which will be the topic of a great deal of discussion in later chapters of this book. The defining characteristic of scientific research is the scientific method (summarized in Rapid Reference 1.2). First described by the En- glish philosopher and scientist Roger Bacon in the 13th century, it is still generally agreed that the scientific method is the basis for all scientific in- vestigation. The scientific method is best thought of as an approach to the acquisition of new knowledge, and this approach effectively distinguishes science from nonscience. To be clear, the scientific method is not actually a single method, as the name would erroneously lead one to believe, but rather an overarching perspective on how scientific investigations should proceed. It is a set of research principles and methods that helps re- searchers obtain valid results from their research studies. Because the sci- entific method deals with the general approach to research rather than the content of specific research studies, it is used by researchers in all different scientific disciplines. As will be seen in the following sections, the biggest benefit of the scientific method is that it provides a set of clear and agreed- upon guidelines for gathering, evaluating, and reporting information in the context of a research study (Cozby, 1993). INTRODUCTION AND OVERVIEW 5 Rapid Reference 1.2 The Scientific Method The development of the scientific method is usually credited to Roger Bacon, a philosopher and scientist from 13th-century England, although some argue that the Italian scientist Galileo Galilei played an important role in formulating the scientific method. Later contributions to the scien- tific method were made by the philosophers Francis Bacon and René Descartes. Although some disagreement exists regarding the exact char- acteristics of the scientific method, most agree that it is characterized by the following elements: • Empirical approach • Observations • Questions • Hypotheses • Experiments • Analyses • Conclusions • Replication There has been some disagreement among researchers over the years regarding the elements that compose the scientific method. In fact, some researchers have even argued that it is impossible to define a universal ap- proach to scientific investigation. Nevertheless, for over 100 years, the scientific method has been the defining feature of scientific research. Re- searchers generally agree that the scientific method is composed of the following key elements (which will be the focus of the remainder of this chapter): an empirical approach, observations, questions, hypotheses, ex- periments, analyses, conclusions, and replication. Before proceeding any further, one word of caution is necessary. In the brief discussion of the scientific method that follows, we will be introduc- ing several new terms and concepts that are related to research design and methodology. Do not be intimidated if you are unfamiliar with some of the content contained in this discussion. The purpose of the following is simply 6 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY to set the stage for the chapters that follow, and we will be elaborating on each of the terms and concepts throughout the remainder of the book. Empirical Approach The scientific method is firmly based on the empirical approach. The em- pirical approach is an evidence-based approach that relies on direct obser- vation and experimentation in the acquisition of new knowledge (see Kazdin, 2003a). In the empirical approach, scientific decisions are made based on the data derived from direct observation and experimentation. Contrast this approach to decision making with the way that most nonsci- entific decisions are made in our daily lives. For example, we have all made decisions based on feelings, hunches, or “gut” instinct. Additionally, we may often reach conclusions or make decisions that are not necessarily based on data, but rather on opinions, speculation, and a hope for the best. The empirical approach, with its emphasis on direct, systematic, and care- ful observation, is best thought of as the guiding principle behind all re- search conducted in accordance with the scientific method. Observations An important component in any scientific investigation is observation. In this sense, observation refers to two distinct concepts—being aware of the world around us and making careful measurements. Observations of the world around us often give rise to the questions that are addressed through scientific research. For example, the Newtonian observation that apples fall from trees stimulated much research into the effects of gravity. There- fore, a keen eye to your surroundings can often provide you with many ideas for research studies. We will discuss the generation of research ideas in more detail in Chapter 2. In the context of science, observation means more than just observing the world around us to get ideas for research. Observation also refers to the process of making careful and accurate measurements, which is a distin- guishing feature of well-conducted scientific investigations. When making INTRODUCTION AND OVERVIEW 7 measurements in the context of research, scientists typically take great precautions to avoid making biased observations. For example, if a re- searcher is observing the amount of time that passes between two events, such as the length of time that elapses between lightning and thunder, it would certainly be advisable for the researcher to use a measurement de- vice that has a high degree of accuracy and reliability. Rather than simply trying to “guesstimate” the amount of time that elapsed between those two events, the researcher would be advised to use a stopwatch or similar measurement device. By doing so, the researcher ensures that the mea- surement is accurate and not biased by extraneous factors. Most people would likely agree that the observations that we make in our daily lives are rarely made so carefully or systematically. An important aspect of measurement is an operational definition. Re- searchers define key concepts and terms in the context of their research studies by using operational definitions. By using operational definitions, researchers ensure that everyone is talking about the same phenomenon. For example, if a researcher wants to study the effects of exercise on stress levels, it would be necessary for the researcher to define what “exercise” is. Does exercise refer to jogging, weight lifting, swimming, jumping rope, or all of the above? By defining “exercise” for the purposes of the study, the researcher makes sure that everyone is referring to the same thing. Clearly, the definition of “exercise” can differ from one study to another, so it is crucial that the researcher define “exercise” in a precise manner in the context of his or her study. Having a clear definition of terms also ensures that the researcher’s study can be replicated by other researchers. The importance of operational definitions will be discussed further in Chapter 2. Questions After getting a research idea, perhaps from making observations of the world around us, the next step in the research process involves translating that research idea into an answerable question. The term “answerable” is particularly important in this respect, and it should not be overlooked. It 8 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY would obviously be a frustrating and ultimately unrewarding endeavor to attempt to answer an unanswerable research question through scientific investigation. An example of an unanswerable research question is the fol- lowing: “Is there an exact replica of me in another universe?” Although this is certainly an intriguing question that would likely yield important in- formation, the current state of science cannot provide an answer to that question. It is therefore important to formulate a research question that can be answered through available scientific methods and procedures. One might ask, for example, whether exercising (i.e., perhaps opera- tionally defined as running three times per week for 30 minutes each time) reduces cholesterol levels. This question could be researched and an- swered using established scientific methods. Hypotheses The next step in the scientific method is coming up with a hypothesis, which is simply an educated—and testable—guess about the answer to your research question. A hypothesis is often described as an attempt by the re- searcher to explain the phenomenon of interest. Hypotheses can take var- ious forms, depending on the question being asked and the type of study being conducted (see Rapid Reference 1.3). A key feature of all hypotheses is that each must make a prediction. Re- member that hypotheses are the researcher’s attempt to explain the phe- nomenon being studied, and that explanation should involve a prediction about the variables being studied. These predictions are then tested by gathering and analyzing data, and the hypotheses can either be supported or refuted (falsified; see Rapid Reference 1.4) on the basis of the data. In their simplest forms, hypotheses are typically phrased as “if-then” statements. For example, a researcher may hypothesize that “if people exercise for 30 minutes per day at least three days per week, then their cho- lesterol levels will be reduced.” This hypothesis makes a prediction about the effects of exercising on levels of cholesterol, and the prediction can be tested by gathering and analyzing data. Two types of hypotheses with which you should be familiar are the null INTRODUCTION AND OVERVIEW 9 hypothesis and the alternate (or experimental) hypothesis. The null hypoth- esis always predicts that there will be no differences between the groups be- ing studied. By contrast, the alternate hypothesis predicts that there will be a difference between the groups. In our example, the null hypothesis would predict that the exercise group and the no-exercise group will not differ 10 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY significantly on levels of cholesterol. The alternate hypothesis would pre- dict that the two groups will differ significantly on cholesterol levels. Hy- potheses will be discussed in more detail in Chapter 2. Experiments After articulating the hypothesis, the next step involves actually conduct- ing the experiment (or research study). For example, if the study involves investigating the effects of exercise on levels of cholesterol, the researcher would design and conduct a study that would attempt to address that ques- tion. As previously mentioned, a key aspect of conducting a research study is measuring the phenomenon of interest in an accurate and reliable manner (see Rapid Reference 1.5). In this example, the researcher would collect data on the cholesterol levels of the study participants by using an accurate and reliable measurement device. Then, the researcher would compare the cholesterol levels of the two groups to see if exercise had any effects. Rapid Reference 1.5 Accuracy vs. Reliability When talking about measurement in the context of research, there is an important distinction between being accurate and being reliable. Accuracy refers to whether the measurement is correct, whereas reliability refers to whether the measurement is consistent. An example may help to clarify the distinction. When throwing darts at a dart board, “accuracy” refers to whether the darts are hitting the bull’s eye (an accurate dart thrower will throw darts that hit the bull’s eye).“Reliability,” on the other hand, refers to whether the darts are hitting the same spot (a reliable dart thrower will throw darts that hit the same spot).Therefore, an accurate and reliable dart thrower will consistently throw the darts in the bull’s eye. As may be evident, however, it is possible for the dart thrower to be reliable, but not accurate. For example, the dart thrower may throw all of the darts in the same spot (which demonstrates high reliability), but that spot may not be the bull’s eye (which demonstrates low accuracy). In the context of mea- surement, both accuracy and reliability are equally important. INTRODUCTION AND OVERVIEW 11 Analyses After conducting the study and gathering the data, the next step involves analyzing the data, which generally calls for the use of statistical tech- niques. The type of statistical techniques used by a researcher depends on the design of the study, the type of data being gathered, and the questions being asked. Although a detailed discussion of statistics is beyond the scope of this text, it is important to be aware of the role of statistics in con- ducting a research study. In short, statistics help researchers minimize the likelihood of reaching an erroneous conclusion about the relationship be- tween the variables being studied. A key decision that researchers must make with the assistance of statis- tics is whether the null hypothesis should be rejected. Remember that the null hypothesis always predicts that there will be no difference between the groups. Therefore, rejecting the null hypothesis means that there is a dif- ference between the groups. In general, most researchers seek to reject the null hypothesis because rejection means the phenomenon being studied (e.g., exercise, medication) had some effect. It is important to note that there are only two choices with respect to the null hypothesis. Specifically, the null hypothesis can be either rejected or not rejected, but it can never be accepted. If we reject the null hypoth- esis, we are concluding that there is a significant difference between the groups. If, however, we do not reject the null hypothesis, then we are con- cluding that we were unable to detect a difference between the groups. To be clear, it does not mean that there is no difference between the two groups. There may in actuality have been a significant difference between the two groups, but we were unable to detect that difference in our study. We will talk more about this important distinction in later chapters. The decision of whether to reject the null hypothesis is based on the results of statistical analyses, and there are two types of errors that re- searchers must be careful to avoid when making this decision—Type I er- rors and Type II errors. A Type I error occurs when a researcher concludes that there is a difference between the groups being studied when, in fact, there is no difference. This is sometimes referred to as a “false positive.” 12 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY By contrast, a Type II error occurs when the researcher concludes that there is not a difference between the two groups being studied when, in fact, there is a difference. This is sometimes referred to as a “false negative.” As previously noted, the conclusion regarding whether there is a difference between the groups is based on the results of statistical analyses. Specifi- cally, with a Type I error, although there is a statistically significant result, it occurred by chance (or error) and there is not actually a difference be- tween the two groups ( Wampold, Davis, & Good, 2003). With a Type II error, there is a nonsignificant statistical result when, in fact, there actually is a difference between the two groups ( Wampold et al.). The typical convention in most fields of science allows for a 5% chance of erroneously rejecting the null hypothesis (i.e., of making a Type I error). In other words, a researcher will conclude that there is a significant differ- ence between the groups being studied (i.e., will reject the null hypothesis) only if the chance of being incorrect is less than 5%. For obvious reasons, researchers want to reduce the likelihood of concluding that there is a sig- nificant difference between the groups being studied when, in fact, there is not a difference. The distinction between Type I and Type II errors is very important, although somewhat complicated. An example may help to clarify these terms. In our example, a researcher conducts a study to determine whether a new medication is effective in treating depression. The new medication is given to Group 1, while a placebo medication is given to Group 2. If, at the conclusion of the study, the researcher concludes that there is a signif- icant difference in levels of depression between Groups 1 and 2 when, in fact, there is no difference, the researcher has made a Type I error. In sim- pler terms, the researcher has detected a difference between the groups that in actuality does not exist; the difference between the groups occurred by chance (or error). By contrast, if the researcher concludes that there is no significant difference in levels of depression between Groups 1 and 2 when, in fact, there is a difference, the researcher has made a Type II er- ror. In simpler terms, the researcher has failed to detect a difference that actually exists between the groups. Which type of error is more serious—Type I or Type II? The answer to INTRODUCTION AND OVERVIEW 13 this question often depends on the context in which the errors are made. Let’s use the medical context as an example. If a doctor diagnoses a patient with cancer when, in fact, the patient does not have cancer (i.e., a false pos- itive), the doctor has committed a Type I error. In this situation, it is likely that the erroneous diagnosis will be discovered ( perhaps through a second opinion) and the patient will undoubtedly be relieved. If, however, the doctor gives the patient a clean bill of health when, in fact, the patient ac- tually has cancer (i.e., a false negative), the doctor has committed a Type II error. Most people would likely agree that a Type II error would be more serious in this example because it would prevent the patient from getting necessary medical treatment. You may be wondering why researchers do not simply set up their re- search studies so that there is even less chance of making a Type I error. For example, wouldn’t it make sense for researchers to set up their re- search studies so that the chance of making a Type I error is less than 1% or, better yet, 0%? The reason that researchers do not set up their studies in this manner has to do with the relationship between making Type I er- rors and making Type II errors. Specifically, there is an inverse relationship Researcher’s Conclusion Difference No Difference Difference Correct decision Type I error No difference Type II error Correct decision 14 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY between Type I errors and Type II errors, which means that by decreasing the probability of making a Type I error, the researcher is increasing the probability of making a Type II error. In other words, if a researcher re- duces the probability of making a Type I error from 5% to 1%, there is now an increased probability that the researcher will make a Type II error by failing to detect a difference that actually exists. The 5% level is a stan- dard convention in most fields of research and represents a compromise between making Type I and Type II errors. Conclusions After analyzing the data and determining whether to reject the null hy- pothesis, the researcher is now in a position to draw some conclusions about the results of the study. For example, if the researcher rejected the null hypothesis, the researcher can conclude that the phenomenon being studied had an effect—a statistically significant effect, to be more precise. If the researcher rejects the null hypothesis in our exercise-cholesterol ex- ample, the researcher is concluding that exercise had an effect on levels of cholesterol. It is important that researchers make only those conclusions that can be supported by the data analyses. Going beyond the data is a cardinal sin that researchers must be careful to avoid. For example, if a researcher con- ducted a correlational study and the results indicated that the two things being studied were strongly related, the researcher could not conclude that one thing caused the other. An oft-repeated statement that will be ex- plained in later chapters is that correlation (i.e., a relationship between two things) does not equal causation. In other words, the fact that two things are related does not mean that one caused the other. Replication One of the most important elements of the scientific method is replica- tion. Replication essentially means conducting the same research study a second time with another group of participants to see whether the same INTRODUCTION AND OVERVIEW 15 DON’T FORGET Correlation Does Not Equal Causation Before looking at an example of why correlation does not equal causa- tion, let’s make sure that we understand what a correlation is. A correla- tion is simply a relationship between two things. For example, size and weight are often correlated because there is a relationship between the size of something and its weight. Specifically, bigger things tend to weigh more.The results of correlational studies simply provide researchers with information regarding the relationship between two or more variables, which may serve as the basis for future studies. It is important, however, that researchers interpret this relationship cautiously. For example, if a researcher finds that eating ice cream is correlated with (i.e., related to) higher rates of drowning, the researcher cannot conclude that eating ice cream causes drowning. It may be that another variable is responsible for the higher rates of drowning. For example, most ice cream is eaten in the summer and most swimming occurs in the summer.There- fore, the higher rates of drowning are not caused by eating ice cream, but rather by the increased number of people who swim during the summer. results are obtained (see Kazdin, 1992; Shaughnessy & Zechmeister, 1997). The same researcher may attempt to replicate previously obtained results, or perhaps other researchers may undertake that task. Replication illustrates an important point about scientific research—namely, that re- searchers should avoid drawing broad conclusions based on the results of a single research study because it is always possible that the results of that particular study were an aberration. In other words, it is possible that the results of the research study were obtained by chance or error and, there- fore, that the results may not accurately represent the actual state of things. However, if the results of a research study are obtained a second time (i.e., replicated), the likelihood that the original study’s findings were obtained by chance or error is greatly reduced. The importance of replication in research cannot be overstated. Repli- cation serves several integral purposes, including establishing the reliabil- ity (i.e., consistency) of the research study’s findings and determining 16 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY whether the same results can be obtained with a different group of partic- ipants. This last point refers to whether the results of the original study are generalizable to other groups of research participants. If the results of a study are replicated, the researchers—and the field in which the re- searchers work—can have greater confidence in the reliability and gener- alizability of the original findings. GOALS OF SCIENTIFIC RESEARCH As stated previously, the goals of scientific research, in broad terms, are to answer questions and acquire new knowledge. This is typically accom- plished by conducting research that permits drawing valid inferences about the relationship between two or more variables ( Kazdin, 1992). In later chapters, we discuss the specific techniques that researchers use to ensure that valid inferences can be drawn from their research, and in Rapid References 1.6 and 1.7 we present some research-related terms you should become familiar with. For now, however, our main discussion will focus on the goals of scientific research in more general terms. Most researchers agree that the three general goals of scientific research are description, prediction, and understanding/explanation (Cozby, 1993; Shaughnessy & Zechmeister, 1997). Description Perhaps the most basic and easily understood goal of scientific research is description. In short, description refers to the process of defining, classify- ing, or categorizing phenomena of interest. For example, a researcher may wish to conduct a research study that has the goal of describing the rela- tionship between two things or events, such as the relationship between cardiovascular exercise and levels of cholesterol. Alternatively, a re- searcher may be interested in describing a single phenomenon, such as the effects of stress on decision making. Descriptive research is useful because it can provide important infor- mation regarding the average member of a group. Specifically, by gather- INTRODUCTION AND OVERVIEW 17 Rapid Reference 1.6 Categories of Research There are two broad categories of research with which researchers must be familiar. Quantitative vs. Qualitative • Quantitative research involves studies that make use of statistical analy- ses to obtain their findings. Key features include formal and systematic measurement and the use of statistics. • Qualitative research involves studies that do not attempt to quantify their results through statistical summary or analysis. Qualitative studies typically involve interviews and observations without formal measure- ment. A case study, which is an in-depth examination of one person, is a form of qualitative research. Qualitative research is often used as a source of hypotheses for later testing in quantitative research. Nomothetic vs. Idiographic • The nomothetic approach uses the study of groups to identify general laws that apply to a large group of people.The goal is often to identify the average member of the group being studied or the average perfor- mance of a group member. • The idiographic approach is the study of an individual. An example of the idiographic approach is the aforementioned case study. The choice of which research approaches to use largely depends on the types of questions being asked in the research study, and different fields of research typically rely on different categories of research to achieve their goals. Social science research, for example, typically relies on quantitative research and the nomothetic approach. In other words, social scientists study large groups of people and rely on statistical analyses to obtain their findings.These two broad categories of research will be the primary focus of this book. ing data on a large enough group of people, a researcher can describe the average member, or the average performance of a member, of the partic- ular group being studied. Perhaps a brief example will help clarify what we mean by this. Let’s say a researcher gathers Scholastic Aptitude Test (SAT) scores from the current freshman class at a prestigious university. By 18 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY Rapid Reference 1.7 Sample vs. Population Two key terms that you must be familiar with are “sample” and “popula- tion.”The population is all individuals of interest to the researcher. For ex- ample, a researcher may be interested in studying anxiety among lawyers; in this example, the population is all lawyers. For obvious reasons, re- searchers are typically unable to study the entire population. In this case it would be difficult, if not impossible, to study anxiety among all lawyers. Therefore, researchers typically study a subset of the population, and that subset is called a sample. Because researchers may not be able to study the entire population of in- terest, it is important that the sample be representative of the population from which it was selected. For example, the sample of lawyers the re- searcher studies should be similar to the population of lawyers. If the pop- ulation of lawyers is composed mainly of White men over the age of 35, studying a sample of lawyers composed mainly of Black women under the age of 30 would obviously be problematic because the sample is not rep- resentative of the population. Studying a representative sample permits the researcher to draw valid inferences about the population. In other words, when a researcher uses a representative sample, if something is true of the sample, it is likely also true of the population. using some simple statistical techniques, the researcher would be able to calculate the average SAT score for the current college freshman at the university. This information would likely be informative for high school students who are considering applying for admittance at the university. One example of descriptive research is correlational research. In corre- lational research (as mentioned earlier), the researcher attempts to determine whether there is a relationship—that is, a correlation—between two or more variables (see Rapid Reference 1.8 for two types of correlation). For example, a researcher may wish to determine whether there is a relation- ship between SAT scores and grade-point averages (GPAs) among a sample of college freshmen. The many uses of correlational research will be discussed in later chapters. INTRODUCTION AND OVERVIEW 19 Prediction Another broad goal of research is prediction. Prediction-based research often stems from previously conducted descriptive research. If a re- searcher finds that there is a relationship (i.e., correlation) between two variables, then it may be possible to predict one variable from knowledge of the other variable. For example, if a researcher found that there is a re- lationship between SAT scores and GPAs, knowledge of the SAT scores alone would allow the researcher to predict the associated GPAs. Many important questions in both science and the so-called real world involve predicting one thing based on knowledge of something else. For example, college admissions boards may attempt to predict success in col- lege based on the GPAs and SAT scores of the applicants. Employers may attempt to predict job success based on work samples, test scores, and can- didate interviews. Psychologists may attempt to predict whether a trau- matic life event leads to depression. Medical doctors may attempt to pre- dict what levels of obesity and high blood pressure are associated with cardiovascular disease and stroke. Meteorologists may attempt to predict the amount of rain based on the temperature, barometric pressure, hu- midity, and weather patterns. In each of these examples, a prediction is be- ing made based on existing knowledge of something else. 20 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY Understanding/Explanation Being able to describe something and having the ability to predict one thing based on knowledge of another are important goals of scientific research, but they do not provide researchers with a true understanding of a phenomenon. One could argue that true understanding of a phenome- non is achieved only when researchers successfully identify the cause or causes of the phenomenon. For example, being able to predict a student’s GPA in college based on his or her SAT scores is important and very prac- tical, but there is a limit to that knowledge. The most important limitation is that a relationship between two things does not permit an inference of causality. In other words, the fact that two things are related and knowl- edge of one thing (e.g., SAT scores) leads to an accurate prediction of the other thing (e.g., GPA) does not mean that one thing caused the other. For example, a relationship between SAT scores and freshman GPAs does not mean that the SAT scores caused the freshman-year GPAs. More than likely, the SAT scores are indicative of other things that may be more directly responsible for the GPAs. For example, the students who score high on the SAT may also be the students who spend a lot of time study- ing, and it is likely the amount of time studying that is the cause of a high GPA. The ability of researchers to make valid causal inferences is determined by the type of research designs they use. Correlational research, as previ- ously noted, does not permit researchers to make causal inferences regard- ing the relationship between the two things that are correlated. By contrast, a randomized controlled study, which will be discussed in detail in Chapter 5, permits researchers to make valid cause-and-effect inferences. There are three prerequisites for drawing an inference of causality be- tween two events (see Shaughnessy & Zechmeister, 1997). First, there must be a relationship (i.e., a correlation) between the two events. In other words, the events must covary—as one changes, the other must also change. If two events do not covary, then a researcher cannot conclude that one event caused the other event. For example, if there is no relation- ship between television viewing and deterioration of eyesight, then one INTRODUCTION AND OVERVIEW 21 cannot reasonably conclude that television viewing causes a deterioration of eyesight. Second, one event (the cause) must precede the other event (the effect). This is sometimes referred to as a time-order relationship. This should make intuitive sense. Obviously, if two events occur simultaneously, it cannot be concluded that one event caused the other. Similarly, if the observed effect comes before the presumed cause, it would make little sense to conclude that the cause caused the effect. Third, alternative explanations for the observed relationship must be ruled out. This is where it gets tricky. Stated another way, a causal expla- nation between two events can be accepted only when other possible causes of the observed relationship have been ruled out. An example may help to clarify this last required condition for causality. Let’s say that a researcher is attempting to study the effects of two different psychothera- pies on levels of depression. The researcher first obtains a representative sample of people with the same level of depression (as measured by a valid and reliable measure) and then randomly assigns them to one of two groups. Group 1 will get Therapy A and Group 2 will get Therapy B. The obvious goal is to compare levels of depression in both groups after pro- viding the therapy. It would be unwise in this situation for the researcher to assign all of the participants under age 30 to Group 1 and all of the par- ticipants over age 30 to Group 2: If, at the conclusion of the study, Group 1 and Group 2 differed signifi- cantly in levels of depression, the researcher would be unable to de- termine which variable—type of therapy or age—was responsible for the reduced depression. We would say that this research has been confounded, which means that two variables (in this case, the type of therapy and age) were allowed to vary (or be different) at the same time. Ideally, only the vari- 22 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY able being studied (e.g., the type of therapy) will differ between the two groups. OVERVIEW OF THE BOOK The focus of this book is, obviously, research design and methodology. Although these terms are sometimes incorrectly used interchangeably, they are distinct concepts with well-defined and circumscribed meanings. Therefore, before proceeding any further, it would behoove us to define these terms, at least temporarily. As defined by Kazdin (1992, 2003a), a recognized leader in the field of research, methodology refers to the prin- ciples, procedures, and practices that govern research, whereas research de- sign refers to the plan used to examine the question of interest. “Method- ology” should be thought of as encompassing the entire process of conducting research (i.e., planning and conducting the research study, drawing conclusions, and disseminating the findings). By contrast, “re- search design” refers to the many ways in which research can be con- ducted to answer the question being asked. These concepts will become clearer throughout this book, but it is important that you understand the focus of this book before reading any further. Essentials of Research Design and Methodology succinctly covers all of the major topic areas within research design and methodology. Each chapter in this book covers a specific research-related topic using easy-to- understand language and illustrative examples. The book is not meant, however, to replace the very extensive and comprehensive coverage of re- search issues that can be found in other publications. For those readers who would like a more in-depth understanding of the specific topic areas covered in this book, we would suggest looking to the publications in- cluded in the reference list at the end of this book. Finally, although each chapter builds upon the knowledge obtained from the previous chapters, each chapter can also be used as a stand-alone summary of the important points within that topic area. For this reason, we occasionally cover some of the same material in more than one chapter. The chapters in Essentials of Research Design and Methodology are organized INTRODUCTION AND OVERVIEW 23 in a manner that accurately reflects the logical flow of a research project from development to conclusion. The first three chapters lay the founda- tion for conducting a research project. This chapter introduced you to some of the key concepts relating to science, research design, and method- ology. As will be discussed, at a basic level, the first step in conducting research involves coming up with an idea and translating that idea into a testable question or statement. Chapter 2 discusses these preliminary stages of research, including choosing a research idea, formulating a re- search problem, choosing appropriate independent and dependent vari- ables, and selecting a sample of participants for your study. As every re- searcher knows, coming up with a well-designed research study can be a challenging process, but the importance of that task cannot be overstated. Chapter 3 discusses some of the more common pitfalls faced by re- searchers when thinking about the design of a research study. After a research question has been formulated, researchers must choose a research design, collect and analyze the data, and draw some con- clusions. Chapter 4 will introduce you to the common measurement issues and strategies that must be considered when designing a research study. Chapter 5 will present a concise summary of the most common types of research designs that are available to researchers; as will be discussed, the type of research design chosen for a particular study depends largely on the question being asked. Chapter 6 will focus on one of the most impor- tant considerations in all of research—validity. Put simply, validity refers to the soundness of the research design being used, with high validity typi- cally producing more accurate and meaningful results. Validity comes in many forms, and Chapter 6 will discuss each one and how to maximize it in the course of research. Chapter 7 will introduce you to many of the is- sues faced by researchers when analyzing data and attempting to draw conclusions based on the data. Most research is subject to oversight by one or more ethical review committees, such as a university-based institutional review board. These committees are charged with the important task of reviewing all proposed research studies to ensure that they comply with applicable regulations governing research, which may be established by the university, the city, 24 ESSENTIALS OF RESEARCH DESIGN AND METHODOLOGY the state, or the federal government, depending on the nature of the re- search being conducted. Knowledge of the commonly encountered ethi- cal issues will assist researchers in avoiding ethical violations and resolving ethical dilemmas. To this end, Chapter 8 will focus on the most commonly encountered ethical issues faced by researchers when designing and con- ducting a research study. Among other things, Chapter 8 will focus on the important topic of informed consent to research. Finally, Chapter 9 will present a brief section on the dissemination of research results, including publication in peer-reviewed journals and pre- sentations at professional conferences. Chapter 9 will include a distillation of major principles of research design and methodology that are appli- cable for those conducting research in a variety of capacities and settings. Chapter 9 will conclude by presenting a checklist of the major research- related concepts and considerations covered throughout this book. Before concluding this chapter, one word of caution is necessary re- garding the focus of this book. As stated previously, research studies come in many different forms, depending on the scientific discipline within which the research is being conducted. For example, most research stud- ies in the field of quantum physics take place in a laboratory and do not in- volve human participants. Contrast this with the research studies that are conducted by social scientists, which may often take place in real-world settings and involve human participants. For the sake of clarity, consis- tency, and ease of reading, we thought that it was necessary to narrow the focus of this book to one broad type of research. Therefore, throughout this book, we will focus primarily on empirical research involving human participants, which is most commonly found in the social and behavioral sciences. Focusing on this type of research permits us to explore a wider range of research-related considerations that must be addressed by re- searchers across many scientific disciplines. INTRODUCTION AND OVERVIEW 25 Two s discussed in Chapter 1, engaging in research can be an exciting and rewarding endeavor. Through research, scientists attempt to answer age-old questions, acquire new knowledge, describe how things work, and ultimately improve the way we all live. Despite the excit- ing and rewarding nature of research, deciding to conduct a research study can be intimidating for both inexperienced and experienced researchers alike. Novice researchers are frequently surprised—and often over- whelmed—by the sheer number of decisions that need to be made in the context of a research study. Depending on the scope and complexity of the research study being considered, there are typically dozens of research- related issues that need to be addressed in the planning stage alone. As a result, the early stages of planning a research study can often seem over- whelming for novice researchers with little experience (and even for sea- soned researchers with considerable experience, although they may not always freely admit it). As will become clear throughout this chapter, much of the work in- volved in conducting a research study actually takes place prior to con- ducting the study itself. All too often, novice researchers underestimate the amount of preparatory groundwork that needs to be accomplished prior to collecting any data. Although the preliminary work of getting a re- search study started differs depending on the type of research being con- ducted, there are some research-related issues that are common to most types of research. For example, prior to collecting any data at all, re- searchers must typically identify a topic area of interest, conduct a litera- 26 PLANNING AND DESIGNING A RESEARCH STUDY 27 ture review, formulate a researchable question, articulate hypotheses, de- termine who or what will be studied, identify the independent and depen- dent variables that will be examined in the study, and choose an appropri- ate research methodology. And these are just a few of the more common research-related issues encountered by researchers. Furthermore, de- pending on the context in which the research is taking place, there may be a push to get the research study started sooner rather than later, which may further contribute to the researcher’s feeling overwhelmed during the planning stage of a research study. In addition to these research-related issues, researchers may also need to consider several logistical and administrative issues. Administrative and logistical issues include things such as who is paying for the research, whether research staff need to be hired, where and when the research study will be conducted, and what approvals need to be obtained (and from whom) to conduct the research study. And this is just a small sam- pling of the preliminary issues that researchers need to address during the planning stage of a research study. The purpose of this chapter is to introduce you to this planning stage. Because research studies differ greatly, both in terms of scope and con- tent, this chapter cannot possibly address all of the issues that need to be considered when planning and designing a research study. Instead, this chapter will focus on the research-related issues that are most commonly encountered by researchers in all scientific fields ( particularly those that involve human participants) when planning and designing a research study. In some ways, you can think of this chapter as a checklist of the ma- jor research-related issues that need to be considered during the planning stage. Although some of the topics discussed in this chapter may not be applicable in the context of your particular research, it is important for you to be aware of these issues. After discussing how researchers typically se- lect the topics that they study, this chapter will discuss literature reviews, the formulation of research problems, the development of testable hy- potheses, the identification and operationalization of independent and de- pendent variables, and the selection and assignment of researc