Voice & body language
Anouk de Groot
Inhoud
College 1: Introduction...........................................................................................................................2
College 2: What to measure?.................................................................................................................8
College 3: Accents and prominence......................................................................................................11
College 4: Manipulation of speech.......................................................................................................18
College 5: Chuncking............................................................................................................................22
College 6: Synthesis..............................................................................................................................30
College 7: Confidence marking.............................................................................................................35
College 8: Error handling and feedback................................................................................................45
College 9: Emotion and surprise...........................................................................................................53
College 10: Stance and Irony................................................................................................................64
College 11: Mimicry..............................................................................................................................70
College 12: Metaphors..........................................................................................................................79
,College 1: Introduction
Today’s programme
Introduction of basic concepts
Regarding voice and body language
o How important are voice and body language?
o How do voice and body language interact?
o How do voice and body language develop with age?
o How universal are aspects of voice and body language?
Example of a study
Introduction of basic concepts
The meaning of a sentence
The same sentence can have different extra connotations
A sentence can be used in different ways so that extra meanings are generated (not
expressed through words or syntax)
The extra information can be derived from context or from the way a sentence is
uttered. E.g. ‘my girlfriend is pregnant’
Non-verbal communication
Informal definition: “The ensemble of features that do not determine what you say,
but how you say it”
Two forms:
o Variation in Voice
o Variation in Body Language
Voice
Features that you can “hear” (encoded in the speech signal itself)
o Intonation (speech melody)
o Loudness, energy (laud and fast talking)
o Tempo (talking fast or slow)
o Rhythm (different with different languages, like Italian have a different rhythm
than Dutch)
o Voice quality (warm voice, sharp voice)
o Pauses (‘uuuhm’ then you fill the pause but you don’t really say anything)
Body language
Features that you can “see” (not always encoded in the speech signal)
o Facial expressions
o Gaze patterns (waar je heen kijkt)
o Hand gestures (different among cultures)
o Pointing (to an object, who don’t see it, don’t know it)
o Posture (shy, aggressive)
o Distance (also culturally dependent)
Voice and body language
There has especially been an interest in use of facial expressions; often without
studying auditory prosodic features
, In this series of lectures, focus on the combined use of those 2 sets of features
What can be signalled in one modality is also often cued in the other modality, but
there may be strength differences
How important are voice and body language?
Long time ago
Quintilianus (first century PC): institution oratoria
Rethorics as the art to persuade an audience (e.g. stylistic tricks, ordering of
information)
Rethorics
Rethorics was mainly dealing with oral language
One more important aspect of rethorics is called “pronunciation”, which refer to
various forms of nonverbal communication: intonation, but also body language, facial
expressions, gestures
In a museum: it is painting from a long time ago, but you still recognize the facial
expressions
Ideally, those features should match the content of the spoken utterances (e.g. happy
message with a happy voice and face)
Presidential debates
Current-day politicians are very much aware of the potential importance of
nonverbal communication
Importance of nonverbal features
It is a popular statement to say that nonverbal features account for more than 90%
of the communication.
This can probably be traced back to early experimental work of Mehrabian and
colleagues on the recognition of emotions in stimuli with conflicting cues…
But it is not at all clear whether it generalizes to all aspects of non-verbal
communication
Study of nonverbal features
A relatively new field
Interesting paradox
o Strong intuition that nonverbal features matter a lot
o But we are only beginning to learn how important they really are
The study of nonverbal features has long been hampered by a lack of tools to record,
measure or analyze specific features
How do voice and body language interact?
Multimodality
Our perception of reality is multimodal; our perceptual system integrates/fuses
information coming from different sensory modalities (vision, hearing, touch, taste)
Imagine a dinner, served in:
o A nice restaurant with candle light and romantic music
o A dirty stable with cobwebs and the noise of a ps.
, o The same pasta will taste different
Visual information is very important, specifically signals coming from a speaker’s face
Visual expressions
Multimodal communication has been the most natural form of human
communication for ages
It is a normal situation that a speaker observes his/her addressee, and that the
addressee sees the speaker; spoken communication without contact is a relatively
new phenomenon and still occurs relatively infrequently
o Zoals bellen?
Starting assumption: human beings do not only interact through auditory
information, but also through visual information (in addition to other sensory
information)
Relation between faces and speech
Faces are often observed in combination with speech (talking hands)
When people speak, we often see them speaking
The visual information in the face can have an effect on the way we perceive speech
Ventriloquism effect
You connect what is moving with the sound
Buikspreken effect
General term for the following perceptual effect: when an auditory and visual signal
are offered simultaneously in different locations, we build a perceptual construct
suggesting that the source of the audio is spatially related to the visual signal
Strong effect which humans can hardly suppress
Suggests some form of recalibration
McGurk effect
Ogen dicht: je hoort ba ba
Ogen open: da da
Open-dicht-open: je hoort verschillende dingen
McGurk effect
Discovered by accident when Harry McGurk (psychologist of the University of Surrey)
and his research assistant John MacDonald were studying how children in different
age categories perceive speech
Combined with a movie clip (vision only) of someone uttering ‘ga’ with the speech
sound of someone uttering ‘ba’
Perceived as ‘da’
In addition
Faces have an impact on how we perceive the speech of others
o Cocktail party phenomenon
Je kunt je focussen op 1 persoon ook al is het geluid van iedereen even
hard
o Lipreading
Compensatory effects: when there is noise on the auditory or visual channel
Anouk de Groot
Inhoud
College 1: Introduction...........................................................................................................................2
College 2: What to measure?.................................................................................................................8
College 3: Accents and prominence......................................................................................................11
College 4: Manipulation of speech.......................................................................................................18
College 5: Chuncking............................................................................................................................22
College 6: Synthesis..............................................................................................................................30
College 7: Confidence marking.............................................................................................................35
College 8: Error handling and feedback................................................................................................45
College 9: Emotion and surprise...........................................................................................................53
College 10: Stance and Irony................................................................................................................64
College 11: Mimicry..............................................................................................................................70
College 12: Metaphors..........................................................................................................................79
,College 1: Introduction
Today’s programme
Introduction of basic concepts
Regarding voice and body language
o How important are voice and body language?
o How do voice and body language interact?
o How do voice and body language develop with age?
o How universal are aspects of voice and body language?
Example of a study
Introduction of basic concepts
The meaning of a sentence
The same sentence can have different extra connotations
A sentence can be used in different ways so that extra meanings are generated (not
expressed through words or syntax)
The extra information can be derived from context or from the way a sentence is
uttered. E.g. ‘my girlfriend is pregnant’
Non-verbal communication
Informal definition: “The ensemble of features that do not determine what you say,
but how you say it”
Two forms:
o Variation in Voice
o Variation in Body Language
Voice
Features that you can “hear” (encoded in the speech signal itself)
o Intonation (speech melody)
o Loudness, energy (laud and fast talking)
o Tempo (talking fast or slow)
o Rhythm (different with different languages, like Italian have a different rhythm
than Dutch)
o Voice quality (warm voice, sharp voice)
o Pauses (‘uuuhm’ then you fill the pause but you don’t really say anything)
Body language
Features that you can “see” (not always encoded in the speech signal)
o Facial expressions
o Gaze patterns (waar je heen kijkt)
o Hand gestures (different among cultures)
o Pointing (to an object, who don’t see it, don’t know it)
o Posture (shy, aggressive)
o Distance (also culturally dependent)
Voice and body language
There has especially been an interest in use of facial expressions; often without
studying auditory prosodic features
, In this series of lectures, focus on the combined use of those 2 sets of features
What can be signalled in one modality is also often cued in the other modality, but
there may be strength differences
How important are voice and body language?
Long time ago
Quintilianus (first century PC): institution oratoria
Rethorics as the art to persuade an audience (e.g. stylistic tricks, ordering of
information)
Rethorics
Rethorics was mainly dealing with oral language
One more important aspect of rethorics is called “pronunciation”, which refer to
various forms of nonverbal communication: intonation, but also body language, facial
expressions, gestures
In a museum: it is painting from a long time ago, but you still recognize the facial
expressions
Ideally, those features should match the content of the spoken utterances (e.g. happy
message with a happy voice and face)
Presidential debates
Current-day politicians are very much aware of the potential importance of
nonverbal communication
Importance of nonverbal features
It is a popular statement to say that nonverbal features account for more than 90%
of the communication.
This can probably be traced back to early experimental work of Mehrabian and
colleagues on the recognition of emotions in stimuli with conflicting cues…
But it is not at all clear whether it generalizes to all aspects of non-verbal
communication
Study of nonverbal features
A relatively new field
Interesting paradox
o Strong intuition that nonverbal features matter a lot
o But we are only beginning to learn how important they really are
The study of nonverbal features has long been hampered by a lack of tools to record,
measure or analyze specific features
How do voice and body language interact?
Multimodality
Our perception of reality is multimodal; our perceptual system integrates/fuses
information coming from different sensory modalities (vision, hearing, touch, taste)
Imagine a dinner, served in:
o A nice restaurant with candle light and romantic music
o A dirty stable with cobwebs and the noise of a ps.
, o The same pasta will taste different
Visual information is very important, specifically signals coming from a speaker’s face
Visual expressions
Multimodal communication has been the most natural form of human
communication for ages
It is a normal situation that a speaker observes his/her addressee, and that the
addressee sees the speaker; spoken communication without contact is a relatively
new phenomenon and still occurs relatively infrequently
o Zoals bellen?
Starting assumption: human beings do not only interact through auditory
information, but also through visual information (in addition to other sensory
information)
Relation between faces and speech
Faces are often observed in combination with speech (talking hands)
When people speak, we often see them speaking
The visual information in the face can have an effect on the way we perceive speech
Ventriloquism effect
You connect what is moving with the sound
Buikspreken effect
General term for the following perceptual effect: when an auditory and visual signal
are offered simultaneously in different locations, we build a perceptual construct
suggesting that the source of the audio is spatially related to the visual signal
Strong effect which humans can hardly suppress
Suggests some form of recalibration
McGurk effect
Ogen dicht: je hoort ba ba
Ogen open: da da
Open-dicht-open: je hoort verschillende dingen
McGurk effect
Discovered by accident when Harry McGurk (psychologist of the University of Surrey)
and his research assistant John MacDonald were studying how children in different
age categories perceive speech
Combined with a movie clip (vision only) of someone uttering ‘ga’ with the speech
sound of someone uttering ‘ba’
Perceived as ‘da’
In addition
Faces have an impact on how we perceive the speech of others
o Cocktail party phenomenon
Je kunt je focussen op 1 persoon ook al is het geluid van iedereen even
hard
o Lipreading
Compensatory effects: when there is noise on the auditory or visual channel
