Discussion lecture 1
Paper 1: How to specify the participant group size for usability
studies
Introduction
Specifying the participant group size for a usability study is being debated a lot.
Increasing the group size increases the study’s reliability by also the costs and
duration of the study.
The aim is to explore the optimal group size and thus being able to inform other
project stakeholders of the basis, risks, and implication associated with a
specification of the participant group size.
Discussion
The broad issues
- Tensions in commercial contexts:
There is a tension between the desire for more reliable findings and the
budget and time required for a study. The study design will also be
constrained by organizational or project realities. One should work
together with the stakeholders to reach a study design that is realistic and
optimal for the project as a whole or that has a benefit to a wider project
must be reached. Sometimes conclusions are drawn on unreliable data
since some data is better than no data.
- Application of research literature:
There are many different usability studies and they take place in a wide
variety of contexts. Usability practitioners then should not simply accept
and generally apply headline figures for participant groups sizes quoted in
research articles without questioning it.
Studies related to problem discovery
There are 2 facts to keep in mind here:
- It is not always easy to objectively define and or identify a problem since
problem are a function of the interaction and do not necessarily constitute
a static feature of the interface.
So a feature of a product may be a problem to some users, but not to
others. It can also be a problem on one day, but not on the other. Problems
can be fuzzy and subjective as well
- Simple enumeration of problems is not a useful exercise when ranking the
severity of problems since ranking problems is complex and highly
subjective
Problem discovery level and context criticality
The participant group sizes probably influences the problem discovery level that
a study will achieve. If this is true, group sizes can be specified according to the
probable mean and or the minimum level op problem discovery sought. One
, should then determine what problem discovery level is appropriate for a study.
Thus the optimal group size depends on what problem discovery level is being
sought and this should be driven by the study’s context
Complexity of the study
The optimal group size should be influences by the study’s complexity, with
larger numbers of participants being required for more complex studies. There is
no one size fits all for the optimal group size for usability studies related to
problem discovery. The studies will involve a degree of subjectivity and numeric
values that result are indicative. There are some useful advices to consider
though:
- At the low end of the range, 3 participants can be enough and at the high
end, 20 participants.
- Five participants will probably discover 80% of all the problems. 10
participants will reveal a minimum of 82% of the problems
- There is also research stating that 7 participants can be enough, even if
the study is complex.
Thus 3-20 participants is valid, 5-10 participants is a sensible baseline range and
group size should be increased along with the study’s complexity and criticality
of the context.
Comparative contexts
When the usability of 2 or more interfaces is run, in order to compare them.
Before one can draw conclusions based on this study, one must be certain that
rejecting the H0 is safe. That means that there is no effect found and any
difference between the study groups occurred purely by chance and because the
study participants are only a sample of the wider population who will use the
system.
The significance level of the findings is determined by the following 2 factors:
- The observed effect size: the difference between the mean of the
results for each group and the degree of variation across the results. If this
increases, the significance level does as well.
- The sample size: the number of participants in each study group. If this
increases, so does the significance level.
The typical challenge is to specify a group size that has a reasonable likelihood of
producing statistically significant findings whilst minimizing the amount of time
and cost that is wasted generating redundant area.
Hypothesis tests can fail due to the following types of errors:
- A type 1 error: when H0 is rejected when it is in fact true
- A type 2 error: when H0 is retained when it should be rejected
The power of a statistical test is the probability that a type 2 error and is
influenced by the effect size, the sample size and the significance level.
Paper 1: How to specify the participant group size for usability
studies
Introduction
Specifying the participant group size for a usability study is being debated a lot.
Increasing the group size increases the study’s reliability by also the costs and
duration of the study.
The aim is to explore the optimal group size and thus being able to inform other
project stakeholders of the basis, risks, and implication associated with a
specification of the participant group size.
Discussion
The broad issues
- Tensions in commercial contexts:
There is a tension between the desire for more reliable findings and the
budget and time required for a study. The study design will also be
constrained by organizational or project realities. One should work
together with the stakeholders to reach a study design that is realistic and
optimal for the project as a whole or that has a benefit to a wider project
must be reached. Sometimes conclusions are drawn on unreliable data
since some data is better than no data.
- Application of research literature:
There are many different usability studies and they take place in a wide
variety of contexts. Usability practitioners then should not simply accept
and generally apply headline figures for participant groups sizes quoted in
research articles without questioning it.
Studies related to problem discovery
There are 2 facts to keep in mind here:
- It is not always easy to objectively define and or identify a problem since
problem are a function of the interaction and do not necessarily constitute
a static feature of the interface.
So a feature of a product may be a problem to some users, but not to
others. It can also be a problem on one day, but not on the other. Problems
can be fuzzy and subjective as well
- Simple enumeration of problems is not a useful exercise when ranking the
severity of problems since ranking problems is complex and highly
subjective
Problem discovery level and context criticality
The participant group sizes probably influences the problem discovery level that
a study will achieve. If this is true, group sizes can be specified according to the
probable mean and or the minimum level op problem discovery sought. One
, should then determine what problem discovery level is appropriate for a study.
Thus the optimal group size depends on what problem discovery level is being
sought and this should be driven by the study’s context
Complexity of the study
The optimal group size should be influences by the study’s complexity, with
larger numbers of participants being required for more complex studies. There is
no one size fits all for the optimal group size for usability studies related to
problem discovery. The studies will involve a degree of subjectivity and numeric
values that result are indicative. There are some useful advices to consider
though:
- At the low end of the range, 3 participants can be enough and at the high
end, 20 participants.
- Five participants will probably discover 80% of all the problems. 10
participants will reveal a minimum of 82% of the problems
- There is also research stating that 7 participants can be enough, even if
the study is complex.
Thus 3-20 participants is valid, 5-10 participants is a sensible baseline range and
group size should be increased along with the study’s complexity and criticality
of the context.
Comparative contexts
When the usability of 2 or more interfaces is run, in order to compare them.
Before one can draw conclusions based on this study, one must be certain that
rejecting the H0 is safe. That means that there is no effect found and any
difference between the study groups occurred purely by chance and because the
study participants are only a sample of the wider population who will use the
system.
The significance level of the findings is determined by the following 2 factors:
- The observed effect size: the difference between the mean of the
results for each group and the degree of variation across the results. If this
increases, the significance level does as well.
- The sample size: the number of participants in each study group. If this
increases, so does the significance level.
The typical challenge is to specify a group size that has a reasonable likelihood of
producing statistically significant findings whilst minimizing the amount of time
and cost that is wasted generating redundant area.
Hypothesis tests can fail due to the following types of errors:
- A type 1 error: when H0 is rejected when it is in fact true
- A type 2 error: when H0 is retained when it should be rejected
The power of a statistical test is the probability that a type 2 error and is
influenced by the effect size, the sample size and the significance level.
