Usability and UX Evaluation
Lecture 1
Usability = the user can do what he or she wants to do the way he or she expects to be able
to do it, without hindrance, hesitation or questions
Absence of frustration
Usability is only an issue when it’s absent
Sometimes only visible when innovation takes place, but we need usability
evaluation to get there
3 dimensions of usability:
Effectiveness (task completion)
Efficiency (time spent)
Satisfaction (no discomfort, positive attitude)
Bailey’s human performance model (all these factors are constantly in motion):
User experience phases:
Phase Time Factors
System reliability phase Before 1950s Stability
System performance phase 1950s – 1960s Speed (no ‘users’ yet)
User performance phase 1960s – 1970s Non-specialist users
Usability phase 1980s – 2000s Novice users
User experience phase 2000s - Young and exciting
User experience = a person’s perceptions and responses resulting from the use and/or
anticipated use of a product, system or service. A consequence of a user’s internal state
(predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the
designed system (e.g. complexity, purpose, usability, functionality, etc.) and the context
within which the interaction occurs.
UX is an extension of the ‘satisfaction’ part of usability
UX in time:
,UX evaluation:
Income of your user
Aligning pre-existing personas
Validating assumptions
Adjust experience throughout all phases
Improve the evaluation of the municipality;
Human – Who is the user?
What is my situation and what are my goals?
Context – How did the user get there?
What happened ‘outside’ the form?
Activity – What was he trying to do?
How did this form relate to what I was trying to achieve?
Usefulness = the degree to which a product enables a user to achieve his or her goals and is
an assessment of the user’s willingness to use the product at all
Danger of early usability testing
o Fixing problems instead of expanding solution space
Apply the right methods for the right phase
o Concept drawings do not provide deep insights
o Task-centered evaluations focus on the negative
Considerations:
Planning approach, methods & time
Sample sampling technique, kind of users, size
Ethics content, data transparency, privacy (GDPR)
Five potential research directions:
A more holistic vision for UX evaluation: pragmatic and hedonic
Inspection methods for hedonic attributed: psychophysiology
Core skills needed for evaluation: what do we all need in an interdisciplinary field?
Triangulation = an approach to data collection and analysis that uses multiple methods,
measures, or approaches to look for convergence on product requirements or problem
areas.
, Benefits of triangulation:
More in-depth understanding
More richness, varied set of data
More convincing, persuasive recommendations
Reduce ‘inappropriate certainty’ that not much is wrong with a design
Prioritizing requirements
Sequential triangulation = the quantitative phase of data collection and analysis follows the
qualitative phase of data collection and analysis
Concurrent triangulation = two or more methods used to confirm, cross-validate, or
corroborate findings within a study.
Lecture 2
Five basic types of performance metrics:
Task of success = measures how effectively users are able to complete a given set of
tasks
o Binary success = you complete a task or you didn’t
o Levels of success = useful when there are reasonable shades of gray
associated with task success
Time on task = measures how much time is required to complete a task
Errors = reflect the mistakes made during a task
Efficiency = examining the amount of effort a user expends to complete a task
Learnability = a way to measure how performance improves or fails to improve over
time
Advantages:
Statistically valid
Clear to explain
Easy to compare
But beware of:
Number fetishism
Cause & effect implications: covariates
Task success
For example: Complete the checkout process by reaching the “End of transaction” screen,
having bought the correct Harry Potter book
Types of task failure
Giving up: participants indicate that they would not continue with the task
Moderator ‘calls’ it: the moderator stops the task because it’s clear that the
participant is not making any progress
Too long: completed the task but not in the predefined time period
Wrong: participants thought that they completed the task successfully, but they
actually did not
A lot of this depends on the thoroughness of your setup.
Lecture 1
Usability = the user can do what he or she wants to do the way he or she expects to be able
to do it, without hindrance, hesitation or questions
Absence of frustration
Usability is only an issue when it’s absent
Sometimes only visible when innovation takes place, but we need usability
evaluation to get there
3 dimensions of usability:
Effectiveness (task completion)
Efficiency (time spent)
Satisfaction (no discomfort, positive attitude)
Bailey’s human performance model (all these factors are constantly in motion):
User experience phases:
Phase Time Factors
System reliability phase Before 1950s Stability
System performance phase 1950s – 1960s Speed (no ‘users’ yet)
User performance phase 1960s – 1970s Non-specialist users
Usability phase 1980s – 2000s Novice users
User experience phase 2000s - Young and exciting
User experience = a person’s perceptions and responses resulting from the use and/or
anticipated use of a product, system or service. A consequence of a user’s internal state
(predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the
designed system (e.g. complexity, purpose, usability, functionality, etc.) and the context
within which the interaction occurs.
UX is an extension of the ‘satisfaction’ part of usability
UX in time:
,UX evaluation:
Income of your user
Aligning pre-existing personas
Validating assumptions
Adjust experience throughout all phases
Improve the evaluation of the municipality;
Human – Who is the user?
What is my situation and what are my goals?
Context – How did the user get there?
What happened ‘outside’ the form?
Activity – What was he trying to do?
How did this form relate to what I was trying to achieve?
Usefulness = the degree to which a product enables a user to achieve his or her goals and is
an assessment of the user’s willingness to use the product at all
Danger of early usability testing
o Fixing problems instead of expanding solution space
Apply the right methods for the right phase
o Concept drawings do not provide deep insights
o Task-centered evaluations focus on the negative
Considerations:
Planning approach, methods & time
Sample sampling technique, kind of users, size
Ethics content, data transparency, privacy (GDPR)
Five potential research directions:
A more holistic vision for UX evaluation: pragmatic and hedonic
Inspection methods for hedonic attributed: psychophysiology
Core skills needed for evaluation: what do we all need in an interdisciplinary field?
Triangulation = an approach to data collection and analysis that uses multiple methods,
measures, or approaches to look for convergence on product requirements or problem
areas.
, Benefits of triangulation:
More in-depth understanding
More richness, varied set of data
More convincing, persuasive recommendations
Reduce ‘inappropriate certainty’ that not much is wrong with a design
Prioritizing requirements
Sequential triangulation = the quantitative phase of data collection and analysis follows the
qualitative phase of data collection and analysis
Concurrent triangulation = two or more methods used to confirm, cross-validate, or
corroborate findings within a study.
Lecture 2
Five basic types of performance metrics:
Task of success = measures how effectively users are able to complete a given set of
tasks
o Binary success = you complete a task or you didn’t
o Levels of success = useful when there are reasonable shades of gray
associated with task success
Time on task = measures how much time is required to complete a task
Errors = reflect the mistakes made during a task
Efficiency = examining the amount of effort a user expends to complete a task
Learnability = a way to measure how performance improves or fails to improve over
time
Advantages:
Statistically valid
Clear to explain
Easy to compare
But beware of:
Number fetishism
Cause & effect implications: covariates
Task success
For example: Complete the checkout process by reaching the “End of transaction” screen,
having bought the correct Harry Potter book
Types of task failure
Giving up: participants indicate that they would not continue with the task
Moderator ‘calls’ it: the moderator stops the task because it’s clear that the
participant is not making any progress
Too long: completed the task but not in the predefined time period
Wrong: participants thought that they completed the task successfully, but they
actually did not
A lot of this depends on the thoroughness of your setup.
