Assignment 4 – Signal Detection Theory
Research Toolbox · Part A: Theory (English version)
SDT – Assignment 4 · page 1
,Contents
Learning objectives and structure of this document
PART A – THEORY
1. Decision making: the three models from lecture 3
2. The basic idea of SDT
3. The model: two bell curves
4. The ROC curve
5. Which measure do you use? The overview table (L04, slide 4)
6. Calculating d', c and β with z-scores (L04, slides 19–24)
7. Does the model fit? Equal or unequal variances (L04, slides 25–32)
8. Bayes' rule and odds (L03, slides 50–57)
9. The optimal criterion (L04, slides 33–35)
10. Theoretical minimum and maximum of all measures
11. Excel toolbox
SDT – Assignment 4 · page 2
,Learning objectives and structure of this document
At the end of this assignment, you will be able to:
• use Bayes' rule to compute the appropriate conditional probability
• work with odds and probabilities
• compute the most common SDT measures
• interpret the SDT measures and draw conclusions based on them
• choose the right signal detection model for the problem
• advise on the optimal criterion in an applied setting
About this English version
This is the English translation of Part A – Theory (pages 1–32 of the Dutch document):
everything you need to understand, from the basic idea to Bayes and the optimal criterion.
Explanations that originally only appeared in the answers have been moved to the right place
in the theory.
The Dutch document additionally contains Part B (step-by-step plans for the exam), Part C
(worked answers Assignment 4.1, the TV viewers and brand names), Part D (Assignment 4.2,
the 26 radiologists) and Part E (Assignment 4.3, the luggage screening). Cross-references to
those parts refer to the Dutch document.
All theory and side notes that only came up during the assignments are included here (among
others 4.4–4.6, 7.9–7.10, 8.6, 9.7–9.11, 10 and 11).
Highlighted text (like this) is text that was highlighted in the original document.
SDT – Assignment 4 · page 3
,PART A – THEORY
1. Decision making: the three models from lecture 3
Lecture 3 is about decision making and uses three models:
1. Expected value (E).
2. Signal detection theory (SDT), which this assignment is about.
3. Bayes' rule, for converting probabilities.
1.1 Expected value (E)
The expected value is what you win or lose on average per time, if you were to repeat something
very often. Take a lottery ticket: it costs €2, and you have a 1 in 1,000,000 chance of winning
€1,000,000.
E = 1/1,000,000 × €1,000,000 = €1
Formula: E = Σ (p × v), in other words: for every possible outcome, add up "probability × value".
For a die: 1/6×1 + 1/6×2 + ... + 1/6×6 = 3.5
For someone who makes the same decision very often, E is therefore the right model.
Above all, remember E = Σ p × v, because it comes back with the "optimal criterion" (see
chapter 9).
1.2 Utility and prospect theory
People buy lottery tickets. According to E you lose money on that on average. Not rational.
People take out insurance. If the insurer earns money on it on average, you lose money on it on
average. So according to E, that is not rational either.
Because people do not always decide according to E, there is also utility: the subjective value of
something. Kahneman's prospect theory elaborates on this: people overestimate very small
probabilities (1 in a million feels like "it could just happen"), and losses feel worse than equally large
gains.
You cannot always read off utility; you have to investigate it by looking at what people prefer. But
that is not easy, because people are not always consistent. Scientists want, for example, that if
someone prefers A to B, and B to C, that person also prefers A to C. In practice this sometimes goes
wrong, and then you cannot build a neat scale.
2. The basic idea of SDT
There is always a situation in which something is or is not present, and someone (or something) has
to decide whether it is there. The teacher who has to decide whether a student knows the material,
the psychologist making a diagnosis, the fire alarm that has to go off. All with the same table:
Response "yes" Response "no"
Signal trial (target present) Hit Miss
Noise trial (target absent) False alarm Correct rejection
Two words you need to know:
• A signal trial is a trial in which the signal really is present.
SDT – Assignment 4 · page 4
, • A noise trial is a trial without a signal, only "noise".
(Why noise? Even if a brand was not on TV, it can feel vaguely familiar (you once saw it in the
supermarket). The model calls that feeling of familiarity the intensity: how strongly does this item
"evoke" something in your head?) ???? → see 3.1 below, where this is explained.
2.1 Peter and Ronald: sensitivity and bias
Peter and Ronald are identical twins, equally smart, equally good eyes. Peter has learned to say NO
when in doubt, Ronald to say YES when in doubt. If they take the same test, they still get different
scores: Ronald has more hits, but also more false alarms. Therefore:
SDT measures two things separately:
1. Sensitivity d': how well can you tell signal and noise apart? This is the same for Peter and
Ronald.
2. Criterion c (also called bias): from which intensity onwards do you say "yes"? This differs.
3. The model: two bell curves
3.1 Familiarity is a number: where do the bells come from?
Step 1: familiarity is a number. Imagine that for every brand name you give a score in your head for
how familiar it feels, from very unfamiliar to very familiar. This does not happen consciously, it is a
feeling, but we pretend it is a number.
Step 2: count how often each score occurs. Take all new brand names (the ones that were not on
TV). Count how many had a familiarity of 1, how many a 2, how many a 3, and so on. Put that in a
bar chart.
What do you see? Most new brand names feel somewhat unfamiliar (in the middle of the low side),
a few feel very unfamiliar, and a few still feel quite familiar. The bar chart is high in the middle and
low at the edges. That shape is called a bell (or normal distribution), because it looks like a bell.
Do the same for all old brand names. They feel more familiar on average, so you get the same kind
of shape, but further to the right.
Those are the two bells:
• Noise bell = the bar chart of the new items
• Signal bell = the bar chart of the old items
3.2 The criterion: the line
Imagine that for every item a "familiarity score" comes up in your head. For new items (noise) it is
low on average, for old items (noise + signal) higher on average. But it varies, so you get two normal
SDT – Assignment 4 · page 5
,distributions that partly overlap. Somewhere you draw a line, the criterion: above the line you say
yes, below it no.
Step 3: the line and the area of a bell. The person says yes if a brand name feels familiar enough.
The boundary for that is the line.
The area of a bell is simply: all items in that bar chart. The part to the right of the line are the items
that felt familiar enough, so where he said yes.
• Signal bell, part to the right of the line = old items with yes = H
• Noise bell, part to the right of the line = new items with yes = F
3.3 Hit rate and false alarm rate
• Hit rate (H) = p(yes | old): the probability of yes, given a signal trial
• False alarm rate (F or FA) = p(yes | new): the probability of yes, given a noise trial
H and F together describe someone's performance completely. You calculate everything from those
two numbers.
Example:
SDT – Assignment 4 · page 6
Research Toolbox · Part A: Theory (English version)
SDT – Assignment 4 · page 1
,Contents
Learning objectives and structure of this document
PART A – THEORY
1. Decision making: the three models from lecture 3
2. The basic idea of SDT
3. The model: two bell curves
4. The ROC curve
5. Which measure do you use? The overview table (L04, slide 4)
6. Calculating d', c and β with z-scores (L04, slides 19–24)
7. Does the model fit? Equal or unequal variances (L04, slides 25–32)
8. Bayes' rule and odds (L03, slides 50–57)
9. The optimal criterion (L04, slides 33–35)
10. Theoretical minimum and maximum of all measures
11. Excel toolbox
SDT – Assignment 4 · page 2
,Learning objectives and structure of this document
At the end of this assignment, you will be able to:
• use Bayes' rule to compute the appropriate conditional probability
• work with odds and probabilities
• compute the most common SDT measures
• interpret the SDT measures and draw conclusions based on them
• choose the right signal detection model for the problem
• advise on the optimal criterion in an applied setting
About this English version
This is the English translation of Part A – Theory (pages 1–32 of the Dutch document):
everything you need to understand, from the basic idea to Bayes and the optimal criterion.
Explanations that originally only appeared in the answers have been moved to the right place
in the theory.
The Dutch document additionally contains Part B (step-by-step plans for the exam), Part C
(worked answers Assignment 4.1, the TV viewers and brand names), Part D (Assignment 4.2,
the 26 radiologists) and Part E (Assignment 4.3, the luggage screening). Cross-references to
those parts refer to the Dutch document.
All theory and side notes that only came up during the assignments are included here (among
others 4.4–4.6, 7.9–7.10, 8.6, 9.7–9.11, 10 and 11).
Highlighted text (like this) is text that was highlighted in the original document.
SDT – Assignment 4 · page 3
,PART A – THEORY
1. Decision making: the three models from lecture 3
Lecture 3 is about decision making and uses three models:
1. Expected value (E).
2. Signal detection theory (SDT), which this assignment is about.
3. Bayes' rule, for converting probabilities.
1.1 Expected value (E)
The expected value is what you win or lose on average per time, if you were to repeat something
very often. Take a lottery ticket: it costs €2, and you have a 1 in 1,000,000 chance of winning
€1,000,000.
E = 1/1,000,000 × €1,000,000 = €1
Formula: E = Σ (p × v), in other words: for every possible outcome, add up "probability × value".
For a die: 1/6×1 + 1/6×2 + ... + 1/6×6 = 3.5
For someone who makes the same decision very often, E is therefore the right model.
Above all, remember E = Σ p × v, because it comes back with the "optimal criterion" (see
chapter 9).
1.2 Utility and prospect theory
People buy lottery tickets. According to E you lose money on that on average. Not rational.
People take out insurance. If the insurer earns money on it on average, you lose money on it on
average. So according to E, that is not rational either.
Because people do not always decide according to E, there is also utility: the subjective value of
something. Kahneman's prospect theory elaborates on this: people overestimate very small
probabilities (1 in a million feels like "it could just happen"), and losses feel worse than equally large
gains.
You cannot always read off utility; you have to investigate it by looking at what people prefer. But
that is not easy, because people are not always consistent. Scientists want, for example, that if
someone prefers A to B, and B to C, that person also prefers A to C. In practice this sometimes goes
wrong, and then you cannot build a neat scale.
2. The basic idea of SDT
There is always a situation in which something is or is not present, and someone (or something) has
to decide whether it is there. The teacher who has to decide whether a student knows the material,
the psychologist making a diagnosis, the fire alarm that has to go off. All with the same table:
Response "yes" Response "no"
Signal trial (target present) Hit Miss
Noise trial (target absent) False alarm Correct rejection
Two words you need to know:
• A signal trial is a trial in which the signal really is present.
SDT – Assignment 4 · page 4
, • A noise trial is a trial without a signal, only "noise".
(Why noise? Even if a brand was not on TV, it can feel vaguely familiar (you once saw it in the
supermarket). The model calls that feeling of familiarity the intensity: how strongly does this item
"evoke" something in your head?) ???? → see 3.1 below, where this is explained.
2.1 Peter and Ronald: sensitivity and bias
Peter and Ronald are identical twins, equally smart, equally good eyes. Peter has learned to say NO
when in doubt, Ronald to say YES when in doubt. If they take the same test, they still get different
scores: Ronald has more hits, but also more false alarms. Therefore:
SDT measures two things separately:
1. Sensitivity d': how well can you tell signal and noise apart? This is the same for Peter and
Ronald.
2. Criterion c (also called bias): from which intensity onwards do you say "yes"? This differs.
3. The model: two bell curves
3.1 Familiarity is a number: where do the bells come from?
Step 1: familiarity is a number. Imagine that for every brand name you give a score in your head for
how familiar it feels, from very unfamiliar to very familiar. This does not happen consciously, it is a
feeling, but we pretend it is a number.
Step 2: count how often each score occurs. Take all new brand names (the ones that were not on
TV). Count how many had a familiarity of 1, how many a 2, how many a 3, and so on. Put that in a
bar chart.
What do you see? Most new brand names feel somewhat unfamiliar (in the middle of the low side),
a few feel very unfamiliar, and a few still feel quite familiar. The bar chart is high in the middle and
low at the edges. That shape is called a bell (or normal distribution), because it looks like a bell.
Do the same for all old brand names. They feel more familiar on average, so you get the same kind
of shape, but further to the right.
Those are the two bells:
• Noise bell = the bar chart of the new items
• Signal bell = the bar chart of the old items
3.2 The criterion: the line
Imagine that for every item a "familiarity score" comes up in your head. For new items (noise) it is
low on average, for old items (noise + signal) higher on average. But it varies, so you get two normal
SDT – Assignment 4 · page 5
,distributions that partly overlap. Somewhere you draw a line, the criterion: above the line you say
yes, below it no.
Step 3: the line and the area of a bell. The person says yes if a brand name feels familiar enough.
The boundary for that is the line.
The area of a bell is simply: all items in that bar chart. The part to the right of the line are the items
that felt familiar enough, so where he said yes.
• Signal bell, part to the right of the line = old items with yes = H
• Noise bell, part to the right of the line = new items with yes = F
3.3 Hit rate and false alarm rate
• Hit rate (H) = p(yes | old): the probability of yes, given a signal trial
• False alarm rate (F or FA) = p(yes | new): the probability of yes, given a noise trial
H and F together describe someone's performance completely. You calculate everything from those
two numbers.
Example:
SDT – Assignment 4 · page 6
