Voice & body language
Samenvatting
Anouk de Groot
Inhoud
College 1: Introduction...........................................................................................................................2
College 2: What to measure?.................................................................................................................4
College 3: Accents and prominence........................................................................................................6
College 4: Manipulation of speech.......................................................................................................10
College 5: Chunking..............................................................................................................................12
College 6: Synthesis..............................................................................................................................16
College 7: Confidence marking.............................................................................................................19
College 8: Error handling and feedback................................................................................................24
College 9: Emotion and surprise...........................................................................................................29
College 10: Stance and Irony................................................................................................................35
College 11: Mimicry..............................................................................................................................38
College 12: Metaphors..........................................................................................................................42
,College 1: Introduction
Introduction of basic concepts
Sentence: the same sentence can have different meanings through extra
connotations (that are not expressed through words or syntax) from the context or
from the way a sentence is uttered
Non-verbal communication: how you say something, consist of
o Auditory forms of prosody
Variation in voice: features that you can hear, e.g. speech melody,
intonation, loudness, tempo, rhythm, voice quality, pauses
o Visual forms of prosody
Variation in body language: features that you can see, e.g. facial
expressions, gaze patterns, hand gestures, pointing, posture, distance
o These can be different among cultures
Regarding voice and body language
How important are voice and body language?
o Rethorics: the art to persuade an audience
An important aspect: pronunciation = various forms of nonverbal
communication: voice and body language features
Already used a long time ago
We can still recognize facial expressions on paintings from long ago,
and ideally those features should match the content of the spoken
utterances
o Now also used a lot in presidential debates
o Nonverbal features account for more than 90% of the communication
o The study of nonverbal features is a new field, and have long been hampered
by a lack of tools to record, measure or analyze
How do voice and body language interact?
o Multimodality: our perception of reality is multimodal (vision, hearing, touch,
taste)
o Visual expressions: normally the speaker and addressee see each other
o Spoken communication without contact is new
o Human beings do not only interact through auditory information, but also
through visual information (in addition to other sensory information)
o Relation between faces and speech: when we see a person talking, when we
see their face, the visual information can have an effect on the way we
perceive the spoken information
o E.g. ventriloguism (buikspreker) effect: when an auditory and visual signal are
offered simultaneously in different locations, we build a perceptual construct
suggesting that the source of the audio is spatially related to the visual signal
McGurk effect: Combine a movie clip (vision only) of someone uttering ‘ga’ with the
speech sound of someone uttering ‘ba’ Perceived as ‘da’
o In addition:
Faces have an impact on how we perceive the speech of others
Cocktail party phenomenon: je kunt je op 1 persoon focussen
, Lipreading
Compensatory effects: when there is noise on the auditory or visual
channel
Voice and body language interactions in feedback mechanisms
How do voice and body language develop with age?
o When children grow older, they improve their skills in using and interpreting
verbal and non-verbal communication
o Intonation patterns, rhythm and features of the voice acquired while in
mother’s womb
o Young infants can intimate facial gestures, like tongue protrusion and mouth
opening
o Infants quickly learn to integrate information coming from different modalities
o Younger children are largely egocentric. As the child grows older, they become
more other directed or socially aware, and nonverbal features become more
functional in nature
o Hypothesis: nonverbal features may reveal differences in social awareness
between younger children, older children and adults
How universal are aspects of voice and body language?
o How do we acquire nonverbal features? By nature or nurture?
o Everyone shows facial expressions (even animals and babies and blind people)
o So there is probably a genetic basis for the use of auditory and facial
expressions, especially related to the expression of basic emotions
Example of a study (surprise experiment)
Many studies use a game-based approach (people are unaware of the real purpose of
the study; do not reflect on their non-verbal features)
Experiment: how do speakers behave nonverbally in expected (normal) vs.
unexpected (surprise) contexts?
Participants are believed to take part in a memory experiment and that they are
interested in the effect of context and in the effect of reading aloud
o 1. Participants have to imagine words that fit in a specific context
o 2. Then they get 10 words shown on a screen one by one
o 3. They have to read those words aloud as they appear on the screen
o 4. Then they have to recall as many words as possible
2 experimental context
o Saying the word “liver” in a
Normal context: organs of the human body
Surprise context: favorite food items of Dutch kids
o Combined with other questions
Perception of participants
o Everyone can see and hear the surprise
o Speakers show these nonverbal features spontaneous and automatic
, College 2: What to measure?
Not voice language + body language, but (voice) + (body language)
Terminology
Verbal vs. nonverbal communication
o Verbal = ‘wordy’ = not what this course is about
o Non-verbal = everything that is not expresses with words
o Sign language = verbal, because it is like words
o Body language = not about words
Words = have a form (sound) and meaning
o To read: meaning = a verb takes a subject (the reader) and object (the thing
that is read)
o Words are conventional, because another sound could be paired with the
same meaning
o Sing languages/visual language are also language, they are conventional, not
mimicry, they are not the same
o Some languages have alternatives for words, like morphemes = verbal
Measurements
Speech measurements
Through vocal fold vibrations
Non-verbal information: tenseness of the vocal folds
People can have a higher or lower voice
Speakers can vary pitch within their range, but you also have a physical limit
Pitch = how often the vocal folds vibrate and how you interpret other voices
Filter (not very relevant for non-verbal part)
Vowels (klinkers): position of tongue/jaw changement in the resonance of higher
frequencies (positie van tong/kaak verandert de resonantie van hogere frequenties)
Consonants (medeklinkers): interruption of airflow (sssss of p)
So: some frequency and interruption information is verbal, some non-verbal. It adds more
details on the voice.
Pitch
We hear a frequency continuously, called pitch
Vocal fold vibration has a frequency: F0
There can be no F0, but then we still hear something
There is always pitch
Use/function of pitch
Non-verbal information
Pitch accents on new/given/contractive information
o We help the listener by giving non-verbal information (e.g. pitch rise) on new
information
Pitch rise at the end, or steady drop when asking a question or saying an assertion
o Extra intonation at the end of a question caused by vocal folds vibration
Samenvatting
Anouk de Groot
Inhoud
College 1: Introduction...........................................................................................................................2
College 2: What to measure?.................................................................................................................4
College 3: Accents and prominence........................................................................................................6
College 4: Manipulation of speech.......................................................................................................10
College 5: Chunking..............................................................................................................................12
College 6: Synthesis..............................................................................................................................16
College 7: Confidence marking.............................................................................................................19
College 8: Error handling and feedback................................................................................................24
College 9: Emotion and surprise...........................................................................................................29
College 10: Stance and Irony................................................................................................................35
College 11: Mimicry..............................................................................................................................38
College 12: Metaphors..........................................................................................................................42
,College 1: Introduction
Introduction of basic concepts
Sentence: the same sentence can have different meanings through extra
connotations (that are not expressed through words or syntax) from the context or
from the way a sentence is uttered
Non-verbal communication: how you say something, consist of
o Auditory forms of prosody
Variation in voice: features that you can hear, e.g. speech melody,
intonation, loudness, tempo, rhythm, voice quality, pauses
o Visual forms of prosody
Variation in body language: features that you can see, e.g. facial
expressions, gaze patterns, hand gestures, pointing, posture, distance
o These can be different among cultures
Regarding voice and body language
How important are voice and body language?
o Rethorics: the art to persuade an audience
An important aspect: pronunciation = various forms of nonverbal
communication: voice and body language features
Already used a long time ago
We can still recognize facial expressions on paintings from long ago,
and ideally those features should match the content of the spoken
utterances
o Now also used a lot in presidential debates
o Nonverbal features account for more than 90% of the communication
o The study of nonverbal features is a new field, and have long been hampered
by a lack of tools to record, measure or analyze
How do voice and body language interact?
o Multimodality: our perception of reality is multimodal (vision, hearing, touch,
taste)
o Visual expressions: normally the speaker and addressee see each other
o Spoken communication without contact is new
o Human beings do not only interact through auditory information, but also
through visual information (in addition to other sensory information)
o Relation between faces and speech: when we see a person talking, when we
see their face, the visual information can have an effect on the way we
perceive the spoken information
o E.g. ventriloguism (buikspreker) effect: when an auditory and visual signal are
offered simultaneously in different locations, we build a perceptual construct
suggesting that the source of the audio is spatially related to the visual signal
McGurk effect: Combine a movie clip (vision only) of someone uttering ‘ga’ with the
speech sound of someone uttering ‘ba’ Perceived as ‘da’
o In addition:
Faces have an impact on how we perceive the speech of others
Cocktail party phenomenon: je kunt je op 1 persoon focussen
, Lipreading
Compensatory effects: when there is noise on the auditory or visual
channel
Voice and body language interactions in feedback mechanisms
How do voice and body language develop with age?
o When children grow older, they improve their skills in using and interpreting
verbal and non-verbal communication
o Intonation patterns, rhythm and features of the voice acquired while in
mother’s womb
o Young infants can intimate facial gestures, like tongue protrusion and mouth
opening
o Infants quickly learn to integrate information coming from different modalities
o Younger children are largely egocentric. As the child grows older, they become
more other directed or socially aware, and nonverbal features become more
functional in nature
o Hypothesis: nonverbal features may reveal differences in social awareness
between younger children, older children and adults
How universal are aspects of voice and body language?
o How do we acquire nonverbal features? By nature or nurture?
o Everyone shows facial expressions (even animals and babies and blind people)
o So there is probably a genetic basis for the use of auditory and facial
expressions, especially related to the expression of basic emotions
Example of a study (surprise experiment)
Many studies use a game-based approach (people are unaware of the real purpose of
the study; do not reflect on their non-verbal features)
Experiment: how do speakers behave nonverbally in expected (normal) vs.
unexpected (surprise) contexts?
Participants are believed to take part in a memory experiment and that they are
interested in the effect of context and in the effect of reading aloud
o 1. Participants have to imagine words that fit in a specific context
o 2. Then they get 10 words shown on a screen one by one
o 3. They have to read those words aloud as they appear on the screen
o 4. Then they have to recall as many words as possible
2 experimental context
o Saying the word “liver” in a
Normal context: organs of the human body
Surprise context: favorite food items of Dutch kids
o Combined with other questions
Perception of participants
o Everyone can see and hear the surprise
o Speakers show these nonverbal features spontaneous and automatic
, College 2: What to measure?
Not voice language + body language, but (voice) + (body language)
Terminology
Verbal vs. nonverbal communication
o Verbal = ‘wordy’ = not what this course is about
o Non-verbal = everything that is not expresses with words
o Sign language = verbal, because it is like words
o Body language = not about words
Words = have a form (sound) and meaning
o To read: meaning = a verb takes a subject (the reader) and object (the thing
that is read)
o Words are conventional, because another sound could be paired with the
same meaning
o Sing languages/visual language are also language, they are conventional, not
mimicry, they are not the same
o Some languages have alternatives for words, like morphemes = verbal
Measurements
Speech measurements
Through vocal fold vibrations
Non-verbal information: tenseness of the vocal folds
People can have a higher or lower voice
Speakers can vary pitch within their range, but you also have a physical limit
Pitch = how often the vocal folds vibrate and how you interpret other voices
Filter (not very relevant for non-verbal part)
Vowels (klinkers): position of tongue/jaw changement in the resonance of higher
frequencies (positie van tong/kaak verandert de resonantie van hogere frequenties)
Consonants (medeklinkers): interruption of airflow (sssss of p)
So: some frequency and interruption information is verbal, some non-verbal. It adds more
details on the voice.
Pitch
We hear a frequency continuously, called pitch
Vocal fold vibration has a frequency: F0
There can be no F0, but then we still hear something
There is always pitch
Use/function of pitch
Non-verbal information
Pitch accents on new/given/contractive information
o We help the listener by giving non-verbal information (e.g. pitch rise) on new
information
Pitch rise at the end, or steady drop when asking a question or saying an assertion
o Extra intonation at the end of a question caused by vocal folds vibration
