Voice and Body Language
Hoorcollege 1
Intrduction of basic concepts
- “My girlfriend is pregnant”
o This sentence has one unique meaning: “my female partner with whom I am not
(yet) married is expecting a child”
- However, the same sentence can have different extra connotations
o [Yessss] My girlfriend is pregnant (joy)
o [Oh, no] My girlfriend is pregnant (fear)
o [What??] My girlfriend is pregnant (surprise)
o [Arghhhh] My girlfriend is pregnant (anger)
In other words: a sentence can be used in different ways so that extra meanings are generated
(not expressed through words of syntax)
Extra information
Can be derived:
- From context (situational, discourse)
- From the way a sentence is uttered
For example, someone’s facial expression makes you interpret it like another connotation
(global)
So: “my girlfriend is pregnant”
Nonverbal communication (NVC): the ensemble of features that do not determine what you
say, but how you say it. Two forms:
1. Variation in voice
2. Variation in body language
Voice: features that you can “hear” (encoded in the speech signal itself)
- Intonation (speech melody)
- Loudness, energy (adaption to the situation)
- Tempo (fast or slow)
- Rhythm (you can’t hear what you speak, but you can recognize that it can’t be some
specific language by hearing the rhythm)
- Voice quality (the way your voice sounds (aggressive, etc.)
- Pauses (filled like uhhmmmm or unfilled)
Body language: features that you can “see” (not always encoded in the speech signal)
- Facial expressions
- Gaze patterns (what/who you’re looking at, the person you’re looking at is the person
you’re speaking to)
- Hand gestures (can be different between cultures)
- Pointing (you don’t have to name it; people will know what you mean by pointing
- Posture (is someone shy?)
- Distance (is it desirable to hug or touch someone? That’s cultural dependent)
1
,How important are voice and body language?
Long time ago
- Quintilanus: institution oratoria
- Rhetoric as the art to persuade an audience (e.g., stylistic tricks, ordering of information)
Rhetoric
- Rhetoric was mainly dealing with oral language
- One important aspect of rhetoric is called “pronunciatio”, which refers to various forms
of nonverbal communication: intonation, but also body language, facial expressions,
gestures, …
- Ideally, those features should match the content of the spoken utterances (e.g., happy
message with a happy voice and face)
Presidential debates
- Current-day politicians are very much aware of the potential importance of nonverbal
communication
Importance of nonverbal features
- It’s a popular statement to say that nonverbal features account for more than 90% of
communication.
- This can probably be traced back to early experimental work of Mehrabian and
colleagues on the recognition of emotions in stimuli with conflicting cues…
- But it’s not at all clear whether it generalizes to all aspects of nonverbal communication.
Study of nonverbal features
- A relatively new field
- Interesting paradox:
o Strong intuition that nonverbal features matter a lot
o But were online beginning to learn how important they really are
- The study of nonverbal features has long been hampered by a lack of tools to record,
measure or analyze specific features
How do voice and body language interact?
Multimodality
- Our perception of reality is multimodal; our perceptual system
integrates/fuses information coming from different sensory modalities
(vision, hearing, touch, taste, ...)
- Imagine a dinner, served in:
o A nice restaurant with candlelight and romantic music
o A dirty stable with cobwebs and the noise of a pneumatic boring machine
- Visual information is very important, specifically signals coming from a
speaker’s face
2
,Visual expressions
- Multimodal communication has been the most natural form of human
communication for ages
- It is a normal situation that a speaker observes his/her addressee, and that the
addressee sees the speaker; spoken communication without visual contact is a relatively
new phenomenon and still occurs relatively infrequently
- Starting assumption: human beings do not only interact through auditory
information, but also through visual information (in addition to other
sensory information)
Relation between faces and speech
- Faces are often observed in combination with speech (talking heads)
- When people speak, we often see them speaking.
- The visual information in the face can have an effect on the way we perceive speech
Ventriloquism effect/Ventriloquism: general term for the following perceptual effect: when an
auditory and visual signal are offered simultaneously in different locations, we build a perceptual
construct suggesting that the source of the audio is spatially related to the visual signal.
- Strong effect which humans can hardly suppress.
- Suggests some form of “recalibration”
McGurk effect: an illusion in which the mismatch between auditory information and visual
information pertaining to a sound’s articulation results in altered perception of that sounds. For
example, when people hear an audio recording of a person uttering the syllable ba while viewing
a video of the speaker uttering ga, they often perceive the syllable as da.
- When auditory /ba/ and visual /ga/ cues point to different speech gestures, we perceive
a blend of these gestures (usually /da/)
- Discovered by accident when Harry McGurk (psychologist of the University of Surrey)
and his research assistant John MacDonald were studying how children in different age
categories perceive speech.
- Combined a movie clip (vision only) of someone uttering /ga/ with the speech sound of
someone uttering /ba/.
o Perceived as /da/.
- Multi-modal speech perception: individuals use both auditory and visual cues when
understanding speech
- Again, very robust effect, which cannot be suppressed.
- Explanation: due to integration of two conflicting modalities
In addition
- Faces have an impact on how we perceive the speech of others
o Cockail party phenomenon
o Lipreading
- Compensatory effects: when there is noise on the auditory or visual channel
- Distribution of vocal and visual cues in human interactions (e.g., feedback mechanism)
3
, How do voice and body language develop with age?
Development of communicative skills
- As children grow older, they improve their skills to communicate with other people
(lexicon, grammar, pronunciation)
- They also become better in using and interpreting nonverbal features
- Defined as the set of features that do not so much determine what people say, but
rather how they say it
- Adults are very skilled users of nonverbal features (see next example)
- How do nonverbal features develop in a growing child?
Roots
- The acquisition of nonverbal features starts very early:
o Intonation patterns, rhythm and features of the voice acquired while in mother’s
womb
o Young infants can imitate facial gestures, like tongue protrusion and mouth
opening
o Infants quickly learn to integrate information coming from different modalities
- Importance of biological and physiological factors, such as:
o Preference for low-ending contours (air pressure)
o Facial expressions similar across the globe (genetically determined)
How about older children?
- But as a child grows older, nonverbal features become more functional in nature
- A child learns to associate specific forms with specific communicative or social functions,
and to manipulate forms
- This is related to the fact that the child becomes more socially aware:
o Increasingly varied environment (family, school, society)
o Becomes more aware of another person’s perspective (Piaget)
- Younger child
o Still largely egocentric
o More other-directed (socially aware)
Social awareness
- In the course of these series of lectures, we will sometimes also look at the extent to
which the use of nonverbal features may differ between different age groups
- Working hypothesis is that nonverbal features may reveal differences in social awareness
between younger children, older children and adults
- See especially lectures on confidence level and expression of emotion
4
Hoorcollege 1
Intrduction of basic concepts
- “My girlfriend is pregnant”
o This sentence has one unique meaning: “my female partner with whom I am not
(yet) married is expecting a child”
- However, the same sentence can have different extra connotations
o [Yessss] My girlfriend is pregnant (joy)
o [Oh, no] My girlfriend is pregnant (fear)
o [What??] My girlfriend is pregnant (surprise)
o [Arghhhh] My girlfriend is pregnant (anger)
In other words: a sentence can be used in different ways so that extra meanings are generated
(not expressed through words of syntax)
Extra information
Can be derived:
- From context (situational, discourse)
- From the way a sentence is uttered
For example, someone’s facial expression makes you interpret it like another connotation
(global)
So: “my girlfriend is pregnant”
Nonverbal communication (NVC): the ensemble of features that do not determine what you
say, but how you say it. Two forms:
1. Variation in voice
2. Variation in body language
Voice: features that you can “hear” (encoded in the speech signal itself)
- Intonation (speech melody)
- Loudness, energy (adaption to the situation)
- Tempo (fast or slow)
- Rhythm (you can’t hear what you speak, but you can recognize that it can’t be some
specific language by hearing the rhythm)
- Voice quality (the way your voice sounds (aggressive, etc.)
- Pauses (filled like uhhmmmm or unfilled)
Body language: features that you can “see” (not always encoded in the speech signal)
- Facial expressions
- Gaze patterns (what/who you’re looking at, the person you’re looking at is the person
you’re speaking to)
- Hand gestures (can be different between cultures)
- Pointing (you don’t have to name it; people will know what you mean by pointing
- Posture (is someone shy?)
- Distance (is it desirable to hug or touch someone? That’s cultural dependent)
1
,How important are voice and body language?
Long time ago
- Quintilanus: institution oratoria
- Rhetoric as the art to persuade an audience (e.g., stylistic tricks, ordering of information)
Rhetoric
- Rhetoric was mainly dealing with oral language
- One important aspect of rhetoric is called “pronunciatio”, which refers to various forms
of nonverbal communication: intonation, but also body language, facial expressions,
gestures, …
- Ideally, those features should match the content of the spoken utterances (e.g., happy
message with a happy voice and face)
Presidential debates
- Current-day politicians are very much aware of the potential importance of nonverbal
communication
Importance of nonverbal features
- It’s a popular statement to say that nonverbal features account for more than 90% of
communication.
- This can probably be traced back to early experimental work of Mehrabian and
colleagues on the recognition of emotions in stimuli with conflicting cues…
- But it’s not at all clear whether it generalizes to all aspects of nonverbal communication.
Study of nonverbal features
- A relatively new field
- Interesting paradox:
o Strong intuition that nonverbal features matter a lot
o But were online beginning to learn how important they really are
- The study of nonverbal features has long been hampered by a lack of tools to record,
measure or analyze specific features
How do voice and body language interact?
Multimodality
- Our perception of reality is multimodal; our perceptual system
integrates/fuses information coming from different sensory modalities
(vision, hearing, touch, taste, ...)
- Imagine a dinner, served in:
o A nice restaurant with candlelight and romantic music
o A dirty stable with cobwebs and the noise of a pneumatic boring machine
- Visual information is very important, specifically signals coming from a
speaker’s face
2
,Visual expressions
- Multimodal communication has been the most natural form of human
communication for ages
- It is a normal situation that a speaker observes his/her addressee, and that the
addressee sees the speaker; spoken communication without visual contact is a relatively
new phenomenon and still occurs relatively infrequently
- Starting assumption: human beings do not only interact through auditory
information, but also through visual information (in addition to other
sensory information)
Relation between faces and speech
- Faces are often observed in combination with speech (talking heads)
- When people speak, we often see them speaking.
- The visual information in the face can have an effect on the way we perceive speech
Ventriloquism effect/Ventriloquism: general term for the following perceptual effect: when an
auditory and visual signal are offered simultaneously in different locations, we build a perceptual
construct suggesting that the source of the audio is spatially related to the visual signal.
- Strong effect which humans can hardly suppress.
- Suggests some form of “recalibration”
McGurk effect: an illusion in which the mismatch between auditory information and visual
information pertaining to a sound’s articulation results in altered perception of that sounds. For
example, when people hear an audio recording of a person uttering the syllable ba while viewing
a video of the speaker uttering ga, they often perceive the syllable as da.
- When auditory /ba/ and visual /ga/ cues point to different speech gestures, we perceive
a blend of these gestures (usually /da/)
- Discovered by accident when Harry McGurk (psychologist of the University of Surrey)
and his research assistant John MacDonald were studying how children in different age
categories perceive speech.
- Combined a movie clip (vision only) of someone uttering /ga/ with the speech sound of
someone uttering /ba/.
o Perceived as /da/.
- Multi-modal speech perception: individuals use both auditory and visual cues when
understanding speech
- Again, very robust effect, which cannot be suppressed.
- Explanation: due to integration of two conflicting modalities
In addition
- Faces have an impact on how we perceive the speech of others
o Cockail party phenomenon
o Lipreading
- Compensatory effects: when there is noise on the auditory or visual channel
- Distribution of vocal and visual cues in human interactions (e.g., feedback mechanism)
3
, How do voice and body language develop with age?
Development of communicative skills
- As children grow older, they improve their skills to communicate with other people
(lexicon, grammar, pronunciation)
- They also become better in using and interpreting nonverbal features
- Defined as the set of features that do not so much determine what people say, but
rather how they say it
- Adults are very skilled users of nonverbal features (see next example)
- How do nonverbal features develop in a growing child?
Roots
- The acquisition of nonverbal features starts very early:
o Intonation patterns, rhythm and features of the voice acquired while in mother’s
womb
o Young infants can imitate facial gestures, like tongue protrusion and mouth
opening
o Infants quickly learn to integrate information coming from different modalities
- Importance of biological and physiological factors, such as:
o Preference for low-ending contours (air pressure)
o Facial expressions similar across the globe (genetically determined)
How about older children?
- But as a child grows older, nonverbal features become more functional in nature
- A child learns to associate specific forms with specific communicative or social functions,
and to manipulate forms
- This is related to the fact that the child becomes more socially aware:
o Increasingly varied environment (family, school, society)
o Becomes more aware of another person’s perspective (Piaget)
- Younger child
o Still largely egocentric
o More other-directed (socially aware)
Social awareness
- In the course of these series of lectures, we will sometimes also look at the extent to
which the use of nonverbal features may differ between different age groups
- Working hypothesis is that nonverbal features may reveal differences in social awareness
between younger children, older children and adults
- See especially lectures on confidence level and expression of emotion
4
