Instructional Design & Evaluation Week 5: Blended learning (learning analytics): conceptualization and
implications
Educational Sciences 2021-2022 Bos, N., Groeneveld, C., van Bruggen, J., & Brand-Gruwel, S. (2016).
The use of recorded lectures in education and the impact on
lecture attendance and exam performance
Week 1: ID&E: Introduction
van Alten, D. C. D., Phielix, C., Janssen, J., Kester, L., (in press).
Byrnes, J. P. (1992). Categorizing and combining theories of
Effects of flipping the classroom on learning outcomes and
cognitive development and learning
satisfaction: A meta-Analysis
Hew, K. F., Lan, M., Tang, Y., Jia, C., & Lo, C. K. (2019). Where is the
Wilson, A., Watson, C., Thompson, T. L., Drew, V., & Doyle, S.
“theory” within the field of educational technology research?
(2017). Learning analytics: Challenges and limitations. Teaching in
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional
Higher Education.
quality of massive open online courses (MOOCs
Week 6: Game-based learning: conceptualization and implications
Week 2: Cognitive load: conceptualization and implications
van Oostendorp, H., Van der Spek, E. D., & Linssen, J. (2014).
Sweller, J., van Merriënboer, J. J. G. & Paas, F. (2019). Cognitive
Adapting the complexity level of a serious game to the proficiency
architecture and instructional design: 20 years later
of players
de Jong, T. (2010). Cognitive load theory, educational research,
Shute, V. J., & Kim, Y. J. (2014). Formative and stealth assessment
and instructional design; some food for thought
Wouters, P., van Nimwegen, C., van Oostendorp, H., & van
Richter, J., Scheiter, K., & Alexander E. (2016). Signaling text-
der Spek, E. D. (2013). A meta-analysis of the cognitive and
picture relations in multimedia learning: A comprehensive meta-
motivational effects of serious games
analysis
Week 7: Motivation: conceptualization and implications
Week 3: Expertise development: conceptualization and implications
Vansteenkiste, M., Sierens, E., Soenens, B., Luyckx, K., & Lens, W.
Ericsson, K. A. (2007). An expert-performance perspective of
(2009). Motivational profiles from a self-determination
research on medical expertise: the study of clinical performance
perspective: The quality of motivation matters
Sternberg, R. J. (1999). Intelligence as developing expertise
de Brabander, C. J., & Glastra, F. J. (2018). Testing a unified model
Dunphy, B. C., & Williamson, S. L. (2004). In pursuit of expertise:
of task-specific motivation: How teachers appraise three
Toward an educational model for expertise development
professional development activities
Lazowski, R. A., & Hulleman, C. S. (2016). Motivation interventions
Week 4: Validity of assessments: conceptualization and implications
in education: A Meta-analytic review
Kane, M. (2004). Certification testing as an illustration of
argument-based validation
Oliveri, M. W., Lawless R. & Mislevy, M. J. (in press). Using
evidence-centered design to support the development of culturally
and linguistically sensitive collaborative problem-solving
assessments
Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A
framework for conceptualizing and evaluating the validity of
instructionally relevant assessments
,🧠 Week 1: Theories of Learning (TOL)
Categorizing and combining theories of cognitive
development and learning (Byrnes, 1992)
🔑Meta-theoretical belief systems (MTBS): philosophies that form the
core of a theory and…
(a) Prompt researchers to investigate certain types of phenomena
(b) Specify the nature of knowledge
(c) Specify the origin of knowledge (epistemology)
Criteria that have to be met to classify a viewpoint on learning as a
theory:
● What is learned - Phenomenon of investigation (e.g., behavior)
● Theoretical models of what is learned - Nature of knowledge:
○ What should be learned? (concepts, procedures, skills),
○ (How) do mental representations causally affect behavior?
(e.g., no, mediate, yes)
● Theoretical mechanisms of how learning takes place - Origin of Implications & Discussion
knowledge: learning mechanism (e.g., innate or ● Guiding theory development.
environment/"formal operations" vs. "operations" vs. "rules") ● Gaining insight into historical linkages between theories.
● Limitation: Shifts in which characteristics/questions are used in
the table to make it fit
Can theories be combined in terms of logical compatibility of their ● Limitation: groups could have also been defined in another way
underlying MTBS? (e.g., cognitivism and behaviourism)
Within a group…
● Depending on the location on the continuum: Extremes cannot be
combined. Middle cannot combine with extremes. Where is the “theory” within the field of
● Depending on theoretical imperialism of the theorist: which educational technology research? (Hew, 2019)
means not possible to combine with other theories. >< (opposite)
to openness to multiple perspectives (eg. Piaget) The article examines if educational technology research uses and applies
● Between groups: Yes, when the theories try to explain different theories by analyzing 503 articles.
phenomena → more combinations can be formed
Types of theories:
● Explanatory: describes the factors affecting a phenomenon (e.g.
CLT)
● Design: how things should be designed to achieve certain goals
(e.g. any instructional design theory)
Results & Discussion:
, ● Most studies don’t explicitly engage with a theory or are vague. 9. Authentic resources: learning resources are drawn from real-
● When they are explicit, they most likely use it to conceptualize the world settings
research, inform data collection, and discuss the results (theory 10. Feedback: learners are given expert feedback on their
exemplification), instead of helping advance the theory with the performance
findings (theory advancement).
● Implications for practice: Results & Discussion:
○ Create “middle-range” theories that can both explain ● None of the MOOCs implemented all 10 principles.
empirical findings in a concrete way and demonstrate the ● Most of them implemented some principles, but poorly.
ability to frame a variety of research topics in the field to ● Potential causes of little/poor implementation:
conceptualize the research design, inform data ○ Instructors and designers don’t have knowledge of ID
manipulation and interpret the results. principles or learning theories
○ Researchers should be more explicity with the underlying ○ They know the principles but only for classroom-use
theories ○ Marketing pressures of the owners of the MOOCs
● The 10 principles are fundamental criteria of instructional quality,
therefore they can be applied to evaluate any form of structured
Instructional quality of massive open online instructional courses rather than only MOOCs and other types of
courses (MOOCs) (Margaryan et al., 2015) online courses, including ‘classroom-only courses.
The article analyzed the quality of 76 MOOC courses using a tool (Course 🥛 Week 2: Cognitive Load Theory
Scan) that assessed the implementation of 10 principles of instruction.
● xMOOCs: hyper-centralised, content-based and linear Cognitive load: Used amount of working memory resources / mental
● cMOOCs: descentralised, non-linear, and focused on exploration effort required to cope with a task
10 principles measured in the Course Scan: Cognitive architecture and ID (Sweller, 2019)
Learning is promoted when…
1. Problem-centred: learners acquire skills in the context of real- Cognitive Load Theory
world problems
2. Activation: learners activate existing knowledge as a foundation 3 types of cognitive load:
3. Demonstration: learners observe a demonstration of the skill to be Intrinsic Extraneous Germane
learned (poor and good practices)
4. Application: learners apply their newly acquired skill to solve Complexity of the Determined by the CL required to learn,
problems information being way the information is which refers to the
5. Integration: learners reflect on, discuss and defend their newly processed, related to presented and what working memory
acquired skill element interactivity the learner is required resources that are
6. Collective knowledge: contribute to collective knowledge to do by the devoted to dealing
7. Collaboration: collaborate with others instructional with intrinsic CL
8. Differentiation: different learners are provided with different procedure
avenues of learning
7 fundamental effects to manage CL:
, 1. Goal-free effect: replace conventional tasks with goal-free tasks
Conceptual criticism Fundamental criticism
that provide learners with a non-specific goal.
•Definitions of types of loads are •Theory adds new ‘effects’ all the
2. Worked example effect: replace conventional tasks with worked
unclear and there’s no consensus time, but ignores or forgets what it
examples that provide learners with a solution they must carefully
•‘Overload’ seems to be ignored. already knows (e.g., schema
study.
•Mental effort and efficiency: theory)
3. Completion problem effect: replace conventional tasks with
completion tasks that provide learners with a partial solution they unclear in terms, calculating •Explaining experimental results in
must complete. efficiency is just wrong terms of CL leads to much
4. Split-attention effect: replace multiple sources of information, •Role of working memory in speculation
distributed either in space (spatial split attention) or time learning is not really defined •Theory seems impossible to
(temporal split attention), with one integrated source of falsify
information.
5. Redundancy effect: replace multiple sources of information that Methodological criticism Practical criticism
are self-contained (they can be understood on their own) with one •Many different scales •Principles sometimes seem to
source of information. •Psychometric properties of the contradict each other
6. Variability effect: replace of series of tasks with similar surface scales: unknown •Different measurement methods:
features with a series of tasks that differ from one another on all •Moment of measuring influences studies are difficult to compare
dimensions on which tasks differ in the real world. outcome: instability of subjective •Cognitive load as an isolated
7. Modality effect: replace a written explanatory text and another rating. theory (no cross-citations)
source of visual information (unimodal) with a spoken explanatory
text and the visual source of information (multimodal).
Signaling text-picture relations in multimedia
4C/ID Model learning (Richter, 2016)
● Meta-analytic review of experimental studies about signaling
● Using signaling led to better performance → more meaningful
learning since it helps the formation of mental representations.
● LPK learners benefited more from the use of signaling than
learners with HPK → LPK learners need this type of instructional
support to allow for comprehension / HPK learners can process
the information without this support
● From other studies: for HPK learners, signaling could be negative
→ they already have related mental representations so the WM
must process unnecessary information when signals are provided
● Other hypothesized moderators (pacing, pictorial format, mapping,
and distinctiveness of signals) were not found as significant
CLT: some food for thought (de Jong, 2010)
🧪 Week 3: Expertise Models
implications
Educational Sciences 2021-2022 Bos, N., Groeneveld, C., van Bruggen, J., & Brand-Gruwel, S. (2016).
The use of recorded lectures in education and the impact on
lecture attendance and exam performance
Week 1: ID&E: Introduction
van Alten, D. C. D., Phielix, C., Janssen, J., Kester, L., (in press).
Byrnes, J. P. (1992). Categorizing and combining theories of
Effects of flipping the classroom on learning outcomes and
cognitive development and learning
satisfaction: A meta-Analysis
Hew, K. F., Lan, M., Tang, Y., Jia, C., & Lo, C. K. (2019). Where is the
Wilson, A., Watson, C., Thompson, T. L., Drew, V., & Doyle, S.
“theory” within the field of educational technology research?
(2017). Learning analytics: Challenges and limitations. Teaching in
Margaryan, A., Bianco, M., & Littlejohn, A. (2015). Instructional
Higher Education.
quality of massive open online courses (MOOCs
Week 6: Game-based learning: conceptualization and implications
Week 2: Cognitive load: conceptualization and implications
van Oostendorp, H., Van der Spek, E. D., & Linssen, J. (2014).
Sweller, J., van Merriënboer, J. J. G. & Paas, F. (2019). Cognitive
Adapting the complexity level of a serious game to the proficiency
architecture and instructional design: 20 years later
of players
de Jong, T. (2010). Cognitive load theory, educational research,
Shute, V. J., & Kim, Y. J. (2014). Formative and stealth assessment
and instructional design; some food for thought
Wouters, P., van Nimwegen, C., van Oostendorp, H., & van
Richter, J., Scheiter, K., & Alexander E. (2016). Signaling text-
der Spek, E. D. (2013). A meta-analysis of the cognitive and
picture relations in multimedia learning: A comprehensive meta-
motivational effects of serious games
analysis
Week 7: Motivation: conceptualization and implications
Week 3: Expertise development: conceptualization and implications
Vansteenkiste, M., Sierens, E., Soenens, B., Luyckx, K., & Lens, W.
Ericsson, K. A. (2007). An expert-performance perspective of
(2009). Motivational profiles from a self-determination
research on medical expertise: the study of clinical performance
perspective: The quality of motivation matters
Sternberg, R. J. (1999). Intelligence as developing expertise
de Brabander, C. J., & Glastra, F. J. (2018). Testing a unified model
Dunphy, B. C., & Williamson, S. L. (2004). In pursuit of expertise:
of task-specific motivation: How teachers appraise three
Toward an educational model for expertise development
professional development activities
Lazowski, R. A., & Hulleman, C. S. (2016). Motivation interventions
Week 4: Validity of assessments: conceptualization and implications
in education: A Meta-analytic review
Kane, M. (2004). Certification testing as an illustration of
argument-based validation
Oliveri, M. W., Lawless R. & Mislevy, M. J. (in press). Using
evidence-centered design to support the development of culturally
and linguistically sensitive collaborative problem-solving
assessments
Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A
framework for conceptualizing and evaluating the validity of
instructionally relevant assessments
,🧠 Week 1: Theories of Learning (TOL)
Categorizing and combining theories of cognitive
development and learning (Byrnes, 1992)
🔑Meta-theoretical belief systems (MTBS): philosophies that form the
core of a theory and…
(a) Prompt researchers to investigate certain types of phenomena
(b) Specify the nature of knowledge
(c) Specify the origin of knowledge (epistemology)
Criteria that have to be met to classify a viewpoint on learning as a
theory:
● What is learned - Phenomenon of investigation (e.g., behavior)
● Theoretical models of what is learned - Nature of knowledge:
○ What should be learned? (concepts, procedures, skills),
○ (How) do mental representations causally affect behavior?
(e.g., no, mediate, yes)
● Theoretical mechanisms of how learning takes place - Origin of Implications & Discussion
knowledge: learning mechanism (e.g., innate or ● Guiding theory development.
environment/"formal operations" vs. "operations" vs. "rules") ● Gaining insight into historical linkages between theories.
● Limitation: Shifts in which characteristics/questions are used in
the table to make it fit
Can theories be combined in terms of logical compatibility of their ● Limitation: groups could have also been defined in another way
underlying MTBS? (e.g., cognitivism and behaviourism)
Within a group…
● Depending on the location on the continuum: Extremes cannot be
combined. Middle cannot combine with extremes. Where is the “theory” within the field of
● Depending on theoretical imperialism of the theorist: which educational technology research? (Hew, 2019)
means not possible to combine with other theories. >< (opposite)
to openness to multiple perspectives (eg. Piaget) The article examines if educational technology research uses and applies
● Between groups: Yes, when the theories try to explain different theories by analyzing 503 articles.
phenomena → more combinations can be formed
Types of theories:
● Explanatory: describes the factors affecting a phenomenon (e.g.
CLT)
● Design: how things should be designed to achieve certain goals
(e.g. any instructional design theory)
Results & Discussion:
, ● Most studies don’t explicitly engage with a theory or are vague. 9. Authentic resources: learning resources are drawn from real-
● When they are explicit, they most likely use it to conceptualize the world settings
research, inform data collection, and discuss the results (theory 10. Feedback: learners are given expert feedback on their
exemplification), instead of helping advance the theory with the performance
findings (theory advancement).
● Implications for practice: Results & Discussion:
○ Create “middle-range” theories that can both explain ● None of the MOOCs implemented all 10 principles.
empirical findings in a concrete way and demonstrate the ● Most of them implemented some principles, but poorly.
ability to frame a variety of research topics in the field to ● Potential causes of little/poor implementation:
conceptualize the research design, inform data ○ Instructors and designers don’t have knowledge of ID
manipulation and interpret the results. principles or learning theories
○ Researchers should be more explicity with the underlying ○ They know the principles but only for classroom-use
theories ○ Marketing pressures of the owners of the MOOCs
● The 10 principles are fundamental criteria of instructional quality,
therefore they can be applied to evaluate any form of structured
Instructional quality of massive open online instructional courses rather than only MOOCs and other types of
courses (MOOCs) (Margaryan et al., 2015) online courses, including ‘classroom-only courses.
The article analyzed the quality of 76 MOOC courses using a tool (Course 🥛 Week 2: Cognitive Load Theory
Scan) that assessed the implementation of 10 principles of instruction.
● xMOOCs: hyper-centralised, content-based and linear Cognitive load: Used amount of working memory resources / mental
● cMOOCs: descentralised, non-linear, and focused on exploration effort required to cope with a task
10 principles measured in the Course Scan: Cognitive architecture and ID (Sweller, 2019)
Learning is promoted when…
1. Problem-centred: learners acquire skills in the context of real- Cognitive Load Theory
world problems
2. Activation: learners activate existing knowledge as a foundation 3 types of cognitive load:
3. Demonstration: learners observe a demonstration of the skill to be Intrinsic Extraneous Germane
learned (poor and good practices)
4. Application: learners apply their newly acquired skill to solve Complexity of the Determined by the CL required to learn,
problems information being way the information is which refers to the
5. Integration: learners reflect on, discuss and defend their newly processed, related to presented and what working memory
acquired skill element interactivity the learner is required resources that are
6. Collective knowledge: contribute to collective knowledge to do by the devoted to dealing
7. Collaboration: collaborate with others instructional with intrinsic CL
8. Differentiation: different learners are provided with different procedure
avenues of learning
7 fundamental effects to manage CL:
, 1. Goal-free effect: replace conventional tasks with goal-free tasks
Conceptual criticism Fundamental criticism
that provide learners with a non-specific goal.
•Definitions of types of loads are •Theory adds new ‘effects’ all the
2. Worked example effect: replace conventional tasks with worked
unclear and there’s no consensus time, but ignores or forgets what it
examples that provide learners with a solution they must carefully
•‘Overload’ seems to be ignored. already knows (e.g., schema
study.
•Mental effort and efficiency: theory)
3. Completion problem effect: replace conventional tasks with
completion tasks that provide learners with a partial solution they unclear in terms, calculating •Explaining experimental results in
must complete. efficiency is just wrong terms of CL leads to much
4. Split-attention effect: replace multiple sources of information, •Role of working memory in speculation
distributed either in space (spatial split attention) or time learning is not really defined •Theory seems impossible to
(temporal split attention), with one integrated source of falsify
information.
5. Redundancy effect: replace multiple sources of information that Methodological criticism Practical criticism
are self-contained (they can be understood on their own) with one •Many different scales •Principles sometimes seem to
source of information. •Psychometric properties of the contradict each other
6. Variability effect: replace of series of tasks with similar surface scales: unknown •Different measurement methods:
features with a series of tasks that differ from one another on all •Moment of measuring influences studies are difficult to compare
dimensions on which tasks differ in the real world. outcome: instability of subjective •Cognitive load as an isolated
7. Modality effect: replace a written explanatory text and another rating. theory (no cross-citations)
source of visual information (unimodal) with a spoken explanatory
text and the visual source of information (multimodal).
Signaling text-picture relations in multimedia
4C/ID Model learning (Richter, 2016)
● Meta-analytic review of experimental studies about signaling
● Using signaling led to better performance → more meaningful
learning since it helps the formation of mental representations.
● LPK learners benefited more from the use of signaling than
learners with HPK → LPK learners need this type of instructional
support to allow for comprehension / HPK learners can process
the information without this support
● From other studies: for HPK learners, signaling could be negative
→ they already have related mental representations so the WM
must process unnecessary information when signals are provided
● Other hypothesized moderators (pacing, pictorial format, mapping,
and distinctiveness of signals) were not found as significant
CLT: some food for thought (de Jong, 2010)
🧪 Week 3: Expertise Models
