Week 1
Evaluation in Human-Computer Interaction over time (based on MacDonald & Atwood(2013))
1. system reliability phase (1940’s - 50’s)
● weinig hele grote machines
● highly trained users → engineers
● main concern of evaluation →
○ minimise system fault time
○ quickly repair errors
2. system performance phase (1950’s - 60’s)
● computers worden meer betrouwbaar
● highly trained users → programmers, computer scientists
● main concern of evaluation →
○ system performance
○ processing speed
○ cost of runtime
3. user performance phase (1960’s - 70’s)
● computer use now includes non-programming
● users → include “non-specialists”
● main concern of evaluation →
○ worker productivity
○ (user) performance-based metrics
4. usability phase (1970’s-2000’s)
● computers become general-purpose powerhouses
● many users → novices
● main concern of evaluation →
○ usability
○ ease of use
○ learnability
5. user experience phase (2000’s - present)
● computers as pervasive & ubiquitous(doordringend en alomvertegenwoordigd)
● users → non-utilitarian use
● main concern of evaluation →
○ expanded to include hedonic factors
experimental research
● historically → scientific principles and laws
● nowadays,also →
○ human thought and behaviour, including in their interaction with computing
systems and technology
○ challenging:
■ high variability across
■ within individuals very difficult to measure
,experiments involve:
● manipulation of one (or more) variables
● attempt to minimise biases, errors and confounds(verwarringen)
● measurement to determine causal connection
○ vs. descriptive goals → accurate depiction/description
○ vs. relational investigation → how are two variables connected(may not be
cause-and-effect)
● pre-defined hypotheses
internal validity
● how likely is the study to determine the intervention’s “true” effect
○ randomization
○ “blinding”
○ valid & appropriate measurement choices for outcomes
○ appropriately sized sample
○ adjustment for confounding variables
external validity
● will the study’s claims hold true in other settings and contexts → outside the tightly
controlled lab design
○ tot op zekere hoogte → lab studies are always more artificial and controlled
than the real world
■ even a well-design study is only a piece of evidence - it is not proof →
importance of replication
defining a hypothesis
● should reference the variables and the relationship between them
● should be precise meaningful, testable, falsifiable
● should be rationalised
○ H1: There is a difference between group/condition A and group/condition B
○ H0: There is no difference between group/condition A and group/condition B
significance
● p = Pr(F | H0 is true)
○ p-value → the probability of the finding if the null hypothesis were true
● f(ES,N)
○ p-value → function of the effect size and the sample size
● p gives only very little information about future p values if the experiment is replicated
, estimation techniques
● effect sizes
○ statistical significance does not mean practical significance
○ effect size required for the latter → size of difference between variables or
extent of variance explained
● confidence intervals
○ range of values you can expect the estimate to fall between if you re-do your
study
○ can calculate this at various probabilities
variables
● independent variable
● dependent variable
research designs
randomised designs
number of IVs/DVs single-factor designs factorial designs
participant-to-condition between-participants design within-participants design
allocation
between-participants / within-participants design
between-subjects design within-subjects design
when there are small individual differences, when there are large individual differences
but large expected differences across
conditions
when learning and carryover effects are when tasks are unlikely to be affected by
likely to influence performance learning and carryover effects are unlikely
to occur
when fatigue may be an issue when working with rare or hard to reach
populations
Evaluation in Human-Computer Interaction over time (based on MacDonald & Atwood(2013))
1. system reliability phase (1940’s - 50’s)
● weinig hele grote machines
● highly trained users → engineers
● main concern of evaluation →
○ minimise system fault time
○ quickly repair errors
2. system performance phase (1950’s - 60’s)
● computers worden meer betrouwbaar
● highly trained users → programmers, computer scientists
● main concern of evaluation →
○ system performance
○ processing speed
○ cost of runtime
3. user performance phase (1960’s - 70’s)
● computer use now includes non-programming
● users → include “non-specialists”
● main concern of evaluation →
○ worker productivity
○ (user) performance-based metrics
4. usability phase (1970’s-2000’s)
● computers become general-purpose powerhouses
● many users → novices
● main concern of evaluation →
○ usability
○ ease of use
○ learnability
5. user experience phase (2000’s - present)
● computers as pervasive & ubiquitous(doordringend en alomvertegenwoordigd)
● users → non-utilitarian use
● main concern of evaluation →
○ expanded to include hedonic factors
experimental research
● historically → scientific principles and laws
● nowadays,also →
○ human thought and behaviour, including in their interaction with computing
systems and technology
○ challenging:
■ high variability across
■ within individuals very difficult to measure
,experiments involve:
● manipulation of one (or more) variables
● attempt to minimise biases, errors and confounds(verwarringen)
● measurement to determine causal connection
○ vs. descriptive goals → accurate depiction/description
○ vs. relational investigation → how are two variables connected(may not be
cause-and-effect)
● pre-defined hypotheses
internal validity
● how likely is the study to determine the intervention’s “true” effect
○ randomization
○ “blinding”
○ valid & appropriate measurement choices for outcomes
○ appropriately sized sample
○ adjustment for confounding variables
external validity
● will the study’s claims hold true in other settings and contexts → outside the tightly
controlled lab design
○ tot op zekere hoogte → lab studies are always more artificial and controlled
than the real world
■ even a well-design study is only a piece of evidence - it is not proof →
importance of replication
defining a hypothesis
● should reference the variables and the relationship between them
● should be precise meaningful, testable, falsifiable
● should be rationalised
○ H1: There is a difference between group/condition A and group/condition B
○ H0: There is no difference between group/condition A and group/condition B
significance
● p = Pr(F | H0 is true)
○ p-value → the probability of the finding if the null hypothesis were true
● f(ES,N)
○ p-value → function of the effect size and the sample size
● p gives only very little information about future p values if the experiment is replicated
, estimation techniques
● effect sizes
○ statistical significance does not mean practical significance
○ effect size required for the latter → size of difference between variables or
extent of variance explained
● confidence intervals
○ range of values you can expect the estimate to fall between if you re-do your
study
○ can calculate this at various probabilities
variables
● independent variable
● dependent variable
research designs
randomised designs
number of IVs/DVs single-factor designs factorial designs
participant-to-condition between-participants design within-participants design
allocation
between-participants / within-participants design
between-subjects design within-subjects design
when there are small individual differences, when there are large individual differences
but large expected differences across
conditions
when learning and carryover effects are when tasks are unlikely to be affected by
likely to influence performance learning and carryover effects are unlikely
to occur
when fatigue may be an issue when working with rare or hard to reach
populations
