A Discovery-Based Workflow for Educational Measurement

2020 PhD in Education, University of California, Santa Barbara 2016 MA in Education, University of California, Santa Barbara 2012 MA in French Civilization & Culture, Middlebury College 2011 BA in Philosophy, Sewanee: the University of the South PUBLICATIONS 2020 Clairmont, Anthony. Wolf, Melissa G., & Maul, Andrew. “The prevention and detection of deception in self-report survey data.” In Basic Elements of Survey Research in Education: Addressing the Problems Your Advisor Never Told You About, eds. Ulemu Luhanga & Gregg Harbaugh. Charlotte, NC: Information Age Publishing. 2020 Arya, Diana, Clairmont, Anthony, Katz, Daniel, & Maul, Andrew. “Measuring Reading Strategy Use.” Educational Measurement (special issue, ed. Michael Kane). 2020 Arya, Diana, Clairmont, Anthony, & Hirsch, Sarah. “Interpreting and explaining data representations: A comparison across grades 1-7.” In Approaches to Lifespan Writing Research: Steps Toward an Actionable Coherence, eds. Ryan Dippre & Talinn Phillips. Colorado: The WAC Clearinghouse, Colorado State University Press, & the University Press of Colorado. 2019 Clairmont, Anthony, & Maul, Andrew. “A Review of Sociocognitive Foundations of Educational Measurement by Robert Mislevy.” Psychometrika, 84(4), . Abstract A Discovery-Based Workflow for Educational Measurement by Anthony Clairmont Educational researchers are often tasked with postulating and measuring unknown processes that account for observed outcomes. Logically, this requires inductive reasoning about which constructs are relevant to the situation we seek to understand, prior to attempts at measurement. Exploratory mixed methods designs, along with a few classic designs from psychometrics, represent the canonical approaches to addressing this research problem. However, I argue that these canonical approaches require rethinking in order to fully embrace both the logic of discovery and the philosophy of measurement. Four normative strands are presented to guide this rethinking: an emphasis on the logic of discovery alongside the customary logic of justification, a move towards radical transparency, an invitation to philosophical exploration, and an appeal to integrate qualitative analysis at every major step of the measurement process. The result of following these normative strands would be a discovery-based workflow for educational measurement. To illustrate such a workflow, I take as an example data from a program evaluation of an intervention for underrepresented minority students in a university STEM program. vi vii Table of contents Acknowledgements iv Vita v Abstract vi Table of Contents vii List of Figures ix List of Tables x Chapter 1: Introduction 1 1.1 Overview and Research Goals 1 1.2 The Case for a Workflow 3 1.3 Common Workflows for Educational Measurement 5 1.3.1 Evidence Centered Design 5 1.3.2 The BEAR Assessment System 7 1.3.3 Sequential mixed-methods designs 8 1.3.4 The Ethnographer’s Toolkit 10 1.4 Limitations of Existing Workflows 11 1.5 Proposed Workflow 17 Chapter 2: Construct Selection 25 2.1 Introduction 25 2.2 Program 26 2.3 Qualitative Methods 28 2.3.1 Method 1: Classroom observation 28 2.3.2 Method 2: Semi-structured group interviews 31 2.4 Qualitative Findings 32 2.5 Discussion 39 2.5.1 Theoretical considerations in construct selection 39 2.5.2 Selection of a focal construct 46 Chapter 3: Construct Definition 56 3.1 Introduction 56 3.2 Habits 59 3.3. Complexity 62 3.4 The Qualifier “Academic” 66 3.5 Ontology of the Construct 68 3.6 Construct Map 71 3.7 Measurability 73 3.8 Normative Argument for Inquiry 77 Chapter 4: Instrumentation 82 4.1 Introduction 82 4.2 Design Principles 82 4.3 Instrument Format 85 4.4 Item Content 89 viii 4.5 Administration 94 4.6 Rasch Measurement Theory 97 4.7 Fitting the Model 101 Chapter 5: Validation 108 5.1 Introduction 108 5.2 Content Validity 110 5.3 Criterion Validity 118 5.4 Response Process 123 5.5 Internal Structure 125 5.6 Consequential Validity 127 5.7 Validity Argument 134 Chapter 6: Conclusion 140 Appendix I: Group Interview Protocol 146 Appendix II: Evidentiary Item Map 147 Appendix III: Item Difficulty, Fit, and DIF in Academic Habit Complexity Scale