Educational effectiveness
Examen: 12/20 punten (25 juni)
Open boek examen (3-5 open vragen) → 5-10 examenvragen worden gegeven in de les
o Vraag 1: je krijgt de resultaten van een studie en je moet je opinie hierover schrijven
o Vraag 2: schrijf een advies voor de minister van onderwijs
Paper: 8/20 punten (15 juni): schrijf een korte academische review over een EER onderwerp
Onderwerpen: Effectiveness criteria, Scientific properties of school effects, Process characteristics,
Methodological issues
o Lijst op Toledo/vrij keuze → Voorbeeld papers op Toledo
3-5 pagina’s lang
Web of Science!
Inhoud: verbondenheid met EER, probleemstelling/onderzoeksvraag, samenvatten van resultaten
(antwoord op onderzoeksvraag), implicaties voor praktijk, beleid, onderzoek, kritische reflectie
General research-questions
What makes a “good” school?
Are there differences between schools in student outcomes?
How large are the differences between schools?
Why do students have more learning gains in school A than in school B? = explaining differences
How can we measure the effectiveness of schools? Effectiveness of teachers? → fair and a valid way?
o Standardized test → do we want to compare teachers?
Is an effective school effective for all subjects? For all students?
Inhoud
1. Introduction to Educational Effectiveness Research (EER)..............................................................................................3
1.1. History of EER..........................................................................................................................................................3
1.2. Educational effectiveness (scheerens, 2004)...........................................................................................................4
2. Five factor model (Edmonds’ Big Five)............................................................................................................................4
3. Model of school learning (Carroll)..................................................................................................................................4
4. Comprehensive model of educational effectiveness......................................................................................................5
5. Integrated multilevel model of education......................................................................................................................5
5.1. Methodology in EER................................................................................................................................................5
5.2. Scientific properties of school effects......................................................................................................................6
6. School effects: Value added models...............................................................................................................................8
6.1. Multilevel structures (= the basic concepts)............................................................................................................8
6.2. Multilevel analysis: Variance decomposition, empty models and value-added models..........................................8
6.3. From an empty multilevel model to a value-added multilevel model.....................................................................9
7. Empty multilevel model (Model 0): model without covariates.....................................................................................10
8. Value added model(s) (model1, model 2): model with covariates...............................................................................11
8.1. Four types of value-added models........................................................................................................................13
8.2. Interpreting the results of a multilevel analysis: an example.................................................................................14
9. International and comparative studies: part 1..............................................................................................................17
9.1. Today’s objectives..................................................................................................................................................17
9.2. Which is the most effective educational system/country?....................................................................................17
9.3. Criticism.................................................................................................................................................................17
9.4. Why international studies......................................................................................................................................22
,10. International and comparative studies: part 2............................................................................................................22
10.1. Text reader: Immigrant children’s educational achievement in Western countries: Origin, destination, and
community effects on mathematical performance. (2008)..........................................................................................22
10.2. Text reader: Educational quasi-markets, school effectiveness and social inequalities.........................................26
11. Absolute versus relative school effects (rankings)......................................................................................................27
12. Online practicum multilevel analyses: exercises.........................................................................................................27
12.1. Exercise 1: variance composition.........................................................................................................................27
12.2. Exercise 2: PIRLS repeat 2018: variance composition and interpreting a multilevel outcome.............................28
12.3. Exercise 3: System level.......................................................................................................................................29
13. Evidence-based education..........................................................................................................................................30
13.1. Evidence based education...................................................................................................................................30
13.2. Why evidence based education?.........................................................................................................................30
13.3. Example 1: successful for all................................................................................................................................30
13.4. Example 2: Grade retention.................................................................................................................................31
13.5. Side-effects of interventions................................................................................................................................31
13.6. Evidence-based education impossible?...............................................................................................................31
13.7. Problems with experimental research.................................................................................................................32
13.8. Evidence-based practice and policy.....................................................................................................................32
13.9. What is essential in education?...........................................................................................................................32
13.10. Monitoring/ data use.........................................................................................................................................33
14. Criticism on EER + Q&A...............................................................................................................................................33
14.1. Methodological criticisms....................................................................................................................................33
14.2. Theoretical criticism............................................................................................................................................33
14.3. Political criticism..................................................................................................................................................34
,1. Introduction to Educational Effectiveness Research (EER)
Tekst uit reader: Educational effectiveness research (EER): a state-of-the-art review. = important
NOT: What makes a “good” school/teacher/system? → effective and less effective schools instead!
o NOT: Are we doing the right things? → teaching right things/ teach the right content
o WEL: Are we doing the things right? → to develop certain competences
Educational quality: EER is 1 perspective to look at this
Educational effectiveness: how process characteristics seen in student outcomes
o We need 2 things for this:
(1) Effectiveness criterion = always student outcomes! Ex. cognitive (average score of math)
Easy to measure and easy to compare
A way to say something about a school/teacher → more/less effective
(2) Effective enhancing conditions
Why there are differences between teachers/schools → Interest in processes
NOT input characteristics: School can’t change: low SES, girls score better on reading
WEL: Process characteristics: elements they can change: instruction methods
1.1. History of EER
Response to Coleman et al. (1966) and Jencks et al. (1972)
Before: School matters! → important wich school you go to
’60-’70
They say: “Schools cannot compensate for society”
o Chances are dependent on the family you are born in
Multilevel models / scientific properties / differential effects
Caused a lot of reaction → EER was born as a reaction
‘80 They want to show that SCHOOLS MATTERS
Differential effects: if a school is effective: is it effective for
every student are specific for some students?
Open the black box of education → related to OV → entering
the right variables
Input/output characteristics → input/process/output
‘90 o Focus on school process: evaluation, policy, …
Importance of class/teacher level
o Focus on school before, here focus on teacher
Large-scale longitudinal studies (LOSO, SiBO, … )
o New types of studies: not 1 measurement anymore
o You can see what the schools adds to the student
> ‘00
International comparative research (PISA, TIMSS, PIRLS, …)
Methodological advances (multivariate ML, SEM,…)
Dynamic nature of education
Examples of an input on the exam!
o Regression coefficients → effect sizes → does the school matter here? → NO
o First model 1: you see student characteristics: all correlated to results of students
Girls score .20 better than boys (references group: here males are not written down)
Matters in which family where you are born (high or low SES) → determined your chances
R²: 20% of differences in achievement scores are explained by students characteristics
o Then model 2: add also school characteristics
Average school SES had a big effect
Student and school characteristics together: not much more explained
o But they are looking at the wrong school characteristics → those are not the effective processes
The teaching matters! → but not inserted in the study
, They did single-level-analysis: they did not take into account that students not all students are
withing the same schools → schools can differ from each others!
o ALL THE ESTIMATES ARE WRONG IN THIS STUDY
1.2. Educational effectiveness (scheerens, 2004)
The most important model
Association between:
o (1) Effectiveness enhancing conditions of schooling
Process characteristics
Important: these characteristics schools can change
Input characteristics (= not manageable): gender, home language, age student/teacher, SES
But important: they influence the process and the outcomes (direct an indirectly)
They explain also differences between students and schools
They need to be taken into account: control for input characteristics
Then you can investigate the true effective!
Context: lager characteristics that have an influence: school ranking, rural settings, …
o (2) effectiveness criterion
Always the students: their outcomes
→ CIPO Model: break open the black box
Effectiveness criteria Theoretical models
1. School results / test results Edmonds (1979)
2. School career (grade repeating, drop-out, …) o Five factor model
3. Metacognitive skills / self-regulated learning Carroll (1963)
4. Student well-being o Model of school learning
5. Motivation / engagement Scheerens (1990)
6. Selfesteem / academic self-concept o CIPO-model
7. Values / attitudes Creemers (1994)
8. Social skills o Comprehensive model of educational
9. Behaviour (truancy, …) effectiveness
10. Long term results (succes in labour market, …) Scheerens (2015):
o Multilevel model of education
11. Learning ga ins
Creemers & Kyriakides (2006):
12. Equity = differences between different types of
o Dynamic model of educational effectiveness
students → high when the gap is small
= the right side of the CIPO model
2. Five factor model (Edmonds’ Big Five)
5 characteristics that are necessary to make you school more effective (= this is confirmed)
o 1. Strong educational leadership
o 2. Emphasis on basic skills acquisition
o 3. Safe and orderly learning climate
o 4. High expectations of pupils’ achievement
o 5. Frequent monitoring of pupils’ progress
3. Model of school learning (Carroll)
Main concept is time (time spend and time need for learning
Achievement is function of 2 things (active and required time)
o Required time: some students need more/less time
Should be as low as possible
Function of 3 things:
Attitude: how good in a course
General ability
Quality of instruction = interest in EER
o High = required time to learn is lower than
Examen: 12/20 punten (25 juni)
Open boek examen (3-5 open vragen) → 5-10 examenvragen worden gegeven in de les
o Vraag 1: je krijgt de resultaten van een studie en je moet je opinie hierover schrijven
o Vraag 2: schrijf een advies voor de minister van onderwijs
Paper: 8/20 punten (15 juni): schrijf een korte academische review over een EER onderwerp
Onderwerpen: Effectiveness criteria, Scientific properties of school effects, Process characteristics,
Methodological issues
o Lijst op Toledo/vrij keuze → Voorbeeld papers op Toledo
3-5 pagina’s lang
Web of Science!
Inhoud: verbondenheid met EER, probleemstelling/onderzoeksvraag, samenvatten van resultaten
(antwoord op onderzoeksvraag), implicaties voor praktijk, beleid, onderzoek, kritische reflectie
General research-questions
What makes a “good” school?
Are there differences between schools in student outcomes?
How large are the differences between schools?
Why do students have more learning gains in school A than in school B? = explaining differences
How can we measure the effectiveness of schools? Effectiveness of teachers? → fair and a valid way?
o Standardized test → do we want to compare teachers?
Is an effective school effective for all subjects? For all students?
Inhoud
1. Introduction to Educational Effectiveness Research (EER)..............................................................................................3
1.1. History of EER..........................................................................................................................................................3
1.2. Educational effectiveness (scheerens, 2004)...........................................................................................................4
2. Five factor model (Edmonds’ Big Five)............................................................................................................................4
3. Model of school learning (Carroll)..................................................................................................................................4
4. Comprehensive model of educational effectiveness......................................................................................................5
5. Integrated multilevel model of education......................................................................................................................5
5.1. Methodology in EER................................................................................................................................................5
5.2. Scientific properties of school effects......................................................................................................................6
6. School effects: Value added models...............................................................................................................................8
6.1. Multilevel structures (= the basic concepts)............................................................................................................8
6.2. Multilevel analysis: Variance decomposition, empty models and value-added models..........................................8
6.3. From an empty multilevel model to a value-added multilevel model.....................................................................9
7. Empty multilevel model (Model 0): model without covariates.....................................................................................10
8. Value added model(s) (model1, model 2): model with covariates...............................................................................11
8.1. Four types of value-added models........................................................................................................................13
8.2. Interpreting the results of a multilevel analysis: an example.................................................................................14
9. International and comparative studies: part 1..............................................................................................................17
9.1. Today’s objectives..................................................................................................................................................17
9.2. Which is the most effective educational system/country?....................................................................................17
9.3. Criticism.................................................................................................................................................................17
9.4. Why international studies......................................................................................................................................22
,10. International and comparative studies: part 2............................................................................................................22
10.1. Text reader: Immigrant children’s educational achievement in Western countries: Origin, destination, and
community effects on mathematical performance. (2008)..........................................................................................22
10.2. Text reader: Educational quasi-markets, school effectiveness and social inequalities.........................................26
11. Absolute versus relative school effects (rankings)......................................................................................................27
12. Online practicum multilevel analyses: exercises.........................................................................................................27
12.1. Exercise 1: variance composition.........................................................................................................................27
12.2. Exercise 2: PIRLS repeat 2018: variance composition and interpreting a multilevel outcome.............................28
12.3. Exercise 3: System level.......................................................................................................................................29
13. Evidence-based education..........................................................................................................................................30
13.1. Evidence based education...................................................................................................................................30
13.2. Why evidence based education?.........................................................................................................................30
13.3. Example 1: successful for all................................................................................................................................30
13.4. Example 2: Grade retention.................................................................................................................................31
13.5. Side-effects of interventions................................................................................................................................31
13.6. Evidence-based education impossible?...............................................................................................................31
13.7. Problems with experimental research.................................................................................................................32
13.8. Evidence-based practice and policy.....................................................................................................................32
13.9. What is essential in education?...........................................................................................................................32
13.10. Monitoring/ data use.........................................................................................................................................33
14. Criticism on EER + Q&A...............................................................................................................................................33
14.1. Methodological criticisms....................................................................................................................................33
14.2. Theoretical criticism............................................................................................................................................33
14.3. Political criticism..................................................................................................................................................34
,1. Introduction to Educational Effectiveness Research (EER)
Tekst uit reader: Educational effectiveness research (EER): a state-of-the-art review. = important
NOT: What makes a “good” school/teacher/system? → effective and less effective schools instead!
o NOT: Are we doing the right things? → teaching right things/ teach the right content
o WEL: Are we doing the things right? → to develop certain competences
Educational quality: EER is 1 perspective to look at this
Educational effectiveness: how process characteristics seen in student outcomes
o We need 2 things for this:
(1) Effectiveness criterion = always student outcomes! Ex. cognitive (average score of math)
Easy to measure and easy to compare
A way to say something about a school/teacher → more/less effective
(2) Effective enhancing conditions
Why there are differences between teachers/schools → Interest in processes
NOT input characteristics: School can’t change: low SES, girls score better on reading
WEL: Process characteristics: elements they can change: instruction methods
1.1. History of EER
Response to Coleman et al. (1966) and Jencks et al. (1972)
Before: School matters! → important wich school you go to
’60-’70
They say: “Schools cannot compensate for society”
o Chances are dependent on the family you are born in
Multilevel models / scientific properties / differential effects
Caused a lot of reaction → EER was born as a reaction
‘80 They want to show that SCHOOLS MATTERS
Differential effects: if a school is effective: is it effective for
every student are specific for some students?
Open the black box of education → related to OV → entering
the right variables
Input/output characteristics → input/process/output
‘90 o Focus on school process: evaluation, policy, …
Importance of class/teacher level
o Focus on school before, here focus on teacher
Large-scale longitudinal studies (LOSO, SiBO, … )
o New types of studies: not 1 measurement anymore
o You can see what the schools adds to the student
> ‘00
International comparative research (PISA, TIMSS, PIRLS, …)
Methodological advances (multivariate ML, SEM,…)
Dynamic nature of education
Examples of an input on the exam!
o Regression coefficients → effect sizes → does the school matter here? → NO
o First model 1: you see student characteristics: all correlated to results of students
Girls score .20 better than boys (references group: here males are not written down)
Matters in which family where you are born (high or low SES) → determined your chances
R²: 20% of differences in achievement scores are explained by students characteristics
o Then model 2: add also school characteristics
Average school SES had a big effect
Student and school characteristics together: not much more explained
o But they are looking at the wrong school characteristics → those are not the effective processes
The teaching matters! → but not inserted in the study
, They did single-level-analysis: they did not take into account that students not all students are
withing the same schools → schools can differ from each others!
o ALL THE ESTIMATES ARE WRONG IN THIS STUDY
1.2. Educational effectiveness (scheerens, 2004)
The most important model
Association between:
o (1) Effectiveness enhancing conditions of schooling
Process characteristics
Important: these characteristics schools can change
Input characteristics (= not manageable): gender, home language, age student/teacher, SES
But important: they influence the process and the outcomes (direct an indirectly)
They explain also differences between students and schools
They need to be taken into account: control for input characteristics
Then you can investigate the true effective!
Context: lager characteristics that have an influence: school ranking, rural settings, …
o (2) effectiveness criterion
Always the students: their outcomes
→ CIPO Model: break open the black box
Effectiveness criteria Theoretical models
1. School results / test results Edmonds (1979)
2. School career (grade repeating, drop-out, …) o Five factor model
3. Metacognitive skills / self-regulated learning Carroll (1963)
4. Student well-being o Model of school learning
5. Motivation / engagement Scheerens (1990)
6. Selfesteem / academic self-concept o CIPO-model
7. Values / attitudes Creemers (1994)
8. Social skills o Comprehensive model of educational
9. Behaviour (truancy, …) effectiveness
10. Long term results (succes in labour market, …) Scheerens (2015):
o Multilevel model of education
11. Learning ga ins
Creemers & Kyriakides (2006):
12. Equity = differences between different types of
o Dynamic model of educational effectiveness
students → high when the gap is small
= the right side of the CIPO model
2. Five factor model (Edmonds’ Big Five)
5 characteristics that are necessary to make you school more effective (= this is confirmed)
o 1. Strong educational leadership
o 2. Emphasis on basic skills acquisition
o 3. Safe and orderly learning climate
o 4. High expectations of pupils’ achievement
o 5. Frequent monitoring of pupils’ progress
3. Model of school learning (Carroll)
Main concept is time (time spend and time need for learning
Achievement is function of 2 things (active and required time)
o Required time: some students need more/less time
Should be as low as possible
Function of 3 things:
Attitude: how good in a course
General ability
Quality of instruction = interest in EER
o High = required time to learn is lower than
