Voice and Body Language – General Introduction
Inhoudsopgave
Voice and Body Language – General Introduction ............................................................................................ 1
Lecture 1: General introduction ..................................................................................................................... 11
Introduction of basic concepts ....................................................................................................................... 11
Connotations .................................................................................................................................................... 11
Non-verbal communication .............................................................................................................................. 11
Definition ..................................................................................................................................................... 11
There are two forms of non-verbal communication ................................................................................... 11
Variation in voice .............................................................................................................................................. 11
Body language .................................................................................................................................................. 12
Features that you can ‘see’ / examples of body language .......................................................................... 12
Voice and Body Language ................................................................................................................................ 13
Regarding Voice and Body Language: How important are voice and body language? .................................... 13
Long time ago… ................................................................................................................................................ 13
Rhetorics ........................................................................................................................................................... 13
Presidential debates ......................................................................................................................................... 14
Importance of nonverbal features .................................................................................................................... 14
Study of nonverbal features ............................................................................................................................. 14
Regarding Voice and Body Language: How do voice and body language interact? ......................................... 14
Multimodality ................................................................................................................................................... 14
Visual expressions............................................................................................................................................. 15
Relation between faces and speech ................................................................................................................. 15
Ventriloquism effect .................................................................................................................................... 15
Ventriloquism ................................................................................................................................................... 15
McGurk effect ................................................................................................................................................... 15
In addition ........................................................................................................................................................ 15
Regarding Voice and Body Language: How do voice and body language develop with age? .......................... 16
Development of communicative skills .............................................................................................................. 16
The adult VS. The child… ................................................................................................................................... 16
Roots................................................................................................................................................................. 16
How about older children? ............................................................................................................................... 17
Younger child .................................................................................................................................................... 17
Older child ........................................................................................................................................................ 17
Social awareness .............................................................................................................................................. 17
1
,Regarding Voice and Body Language: How universal are voice and body language? ...................................... 17
How do we acquire nonverbal features? .......................................................................................................... 17
Apes make facial expressions ........................................................................................................................... 17
Dogs also make expressions ............................................................................................................................. 18
Young kids also make expressions .................................................................................................................... 18
People from New Guinea do it… ....................................................................................................................... 18
Blind people do it… ........................................................................................................................................... 19
So… ................................................................................................................................................................... 19
Example of a study ......................................................................................................................................... 20
Approach .......................................................................................................................................................... 20
Surprise! ........................................................................................................................................................... 20
One experiment ................................................................................................................................................ 20
2. Experimental contexts. ................................................................................................................................. 20
Conclusion of the experiment ........................................................................................................................... 21
Practical information about the course ............................................................................................................ 21
Overview ........................................................................................................................................................... 21
Lecture 2: What to measure for VBL ............................................................................................................... 21
Introduction ...................................................................................................................................................... 21
Terminology...................................................................................................................................................... 22
Verbal vs. non-verbalcommunication.......................................................................................................... 22
Words .......................................................................................................................................................... 22
Words, signs, morphemes ................................................................................................................................ 22
Word alternatives ........................................................................................................................................ 22
Modalities .................................................................................................................................................... 23
First modality: Speech – Source .................................................................................................................. 23
Filter............................................................................................................................................................. 23
Pitch .................................................................................................................................................................. 24
Pitch: definition ........................................................................................................................................... 24
Pitch: where do we use it for / what is its function? ................................................................................... 24
Demonstration............................................................................................................................................. 24
Another one where you can see a pitch accent: ............................................................................................... 26
Body language measures ................................................................................................................................. 27
List of body language variables ........................................................................................................................ 27
Lecture 3: Accents and prominence................................................................................................................ 27
Things we saw previously… .............................................................................................................................. 27
Highlight information structure ........................................................................................................................ 28
Information structure ....................................................................................................................................... 28
Program ............................................................................................................................................................ 29
Accents and prominence ................................................................................................................................ 29
Prominence marking ........................................................................................................................................ 29
2
, Visual cues ........................................................................................................................................................ 30
Obama video .................................................................................................................................................... 30
Link between accents and brows ...................................................................................................................... 30
Production data: newsreaders ......................................................................................................................... 30
Production data (2): beat gestures ................................................................................................................... 30
Conclusion from production data ..................................................................................................................... 31
Current lecture.................................................................................................................................................. 31
Experiment 1 .................................................................................................................................................... 31
Reaction times experiment ............................................................................................................................... 31
Task for subjects ............................................................................................................................................... 32
Congruent!!! ..................................................................................................................................................... 32
Incongruent!!! .................................................................................................................................................. 32
Results .............................................................................................................................................................. 32
Discussion Experiment 1 ................................................................................................................................... 33
Experiment 2 .................................................................................................................................................... 33
Vertical dimension (top to bottom of the face)........................................................................................... 33
Horizontal dimension: the left or right part of the face ................................................................................... 33
Original ............................................................................................................................................................. 34
Prominence detection experiment.................................................................................................................... 34
Task for subjects ............................................................................................................................................... 34
Videos ............................................................................................................................................................... 35
Results horizontal dimension (% correct) ......................................................................................................... 35
Results vertical dimension ................................................................................................................................ 35
Discussion experiment 2 ................................................................................................................................... 35
Crosslinguistic comparisons ........................................................................................................................... 36
Crosslinguistic differences in prosody ............................................................................................................... 36
Accents in different languages ......................................................................................................................... 36
English in South-Africa (Swerts and Zerbian 2010) .......................................................................................... 36
Task .................................................................................................................................................................. 37
Contrast: adjective RED cow Position: final .................................................................................................. 37
Contrast: noun (red COW) Position: final..................................................................................................... 37
Contrast: adjective (RED cow) Position: nonfinal ......................................................................................... 38
Contrast: noun (red COW) Position: nonfinal .............................................................................................. 38
Differences (based on perception results) ........................................................................................................ 38
Implications ...................................................................................................................................................... 38
Conclusion ........................................................................................................................................................ 39
Lecture 4: Manipulation of speech ................................................................................................................. 39
Overview ........................................................................................................................................................... 39
3
, Non-verbal cues ................................................................................................................................................ 39
Research ........................................................................................................................................................... 39
Artificial production ..................................................................................................................................... 39
Prominence types (see also the last lecture) .................................................................................................... 39
Prominence (meaning) types ....................................................................................................................... 40
Prominence types ............................................................................................................................................. 40
Example 1 .................................................................................................................................................... 40
Example 2 .................................................................................................................................................... 40
Example 3 .................................................................................................................................................... 41
Example 4 .................................................................................................................................................... 41
Example 5 .................................................................................................................................................... 41
Example 6 .................................................................................................................................................... 41
Example 7 .................................................................................................................................................... 41
Gives information ............................................................................................................................................. 41
Link ................................................................................................................................................................... 41
What about Marc?....................................................................................................................................... 41
Live demonstration ........................................................................................................................................... 42
Tom showed manipulating speech with the program “Praat” ......................................................................... 42
How can you say things? Speech acts, intent/valence ..................................................................................... 42
Speech acts ....................................................................................................................................................... 42
Intent/valence .................................................................................................................................................. 42
Live demonstration 2 ........................................................................................................................................ 43
Next week(s) ..................................................................................................................................................... 43
Lecture 5: Speech synthesis ............................................................................................................................ 43
Overview ........................................................................................................................................................... 43
Speech synthesis ............................................................................................................................................... 43
State of the art............................................................................................................................................. 43
Forward prediction ........................................................................................................................................... 43
Wavenet synthesis ............................................................................................................................................ 44
Naturalness .................................................................................................................................................. 44
Non-verbal cues ................................................................................................................................................ 44
Present? ....................................................................................................................................................... 44
Meaning/signaling ....................................................................................................................................... 44
More human(e)? ............................................................................................................................................... 45
Importance of affect ......................................................................................................................................... 45
Research questions in James et al. (2020) ................................................................................................... 46
Method ............................................................................................................................................................. 46
Results (1,2) ...................................................................................................................................................... 46
Results (3) ......................................................................................................................................................... 47
Schutz, Schermerhorn and Krahmer (2006) ..................................................................................................... 47
Cooperate with space robot ............................................................................................................................. 47
Method ............................................................................................................................................................. 47
Result ................................................................................................................................................................ 47
4
Inhoudsopgave
Voice and Body Language – General Introduction ............................................................................................ 1
Lecture 1: General introduction ..................................................................................................................... 11
Introduction of basic concepts ....................................................................................................................... 11
Connotations .................................................................................................................................................... 11
Non-verbal communication .............................................................................................................................. 11
Definition ..................................................................................................................................................... 11
There are two forms of non-verbal communication ................................................................................... 11
Variation in voice .............................................................................................................................................. 11
Body language .................................................................................................................................................. 12
Features that you can ‘see’ / examples of body language .......................................................................... 12
Voice and Body Language ................................................................................................................................ 13
Regarding Voice and Body Language: How important are voice and body language? .................................... 13
Long time ago… ................................................................................................................................................ 13
Rhetorics ........................................................................................................................................................... 13
Presidential debates ......................................................................................................................................... 14
Importance of nonverbal features .................................................................................................................... 14
Study of nonverbal features ............................................................................................................................. 14
Regarding Voice and Body Language: How do voice and body language interact? ......................................... 14
Multimodality ................................................................................................................................................... 14
Visual expressions............................................................................................................................................. 15
Relation between faces and speech ................................................................................................................. 15
Ventriloquism effect .................................................................................................................................... 15
Ventriloquism ................................................................................................................................................... 15
McGurk effect ................................................................................................................................................... 15
In addition ........................................................................................................................................................ 15
Regarding Voice and Body Language: How do voice and body language develop with age? .......................... 16
Development of communicative skills .............................................................................................................. 16
The adult VS. The child… ................................................................................................................................... 16
Roots................................................................................................................................................................. 16
How about older children? ............................................................................................................................... 17
Younger child .................................................................................................................................................... 17
Older child ........................................................................................................................................................ 17
Social awareness .............................................................................................................................................. 17
1
,Regarding Voice and Body Language: How universal are voice and body language? ...................................... 17
How do we acquire nonverbal features? .......................................................................................................... 17
Apes make facial expressions ........................................................................................................................... 17
Dogs also make expressions ............................................................................................................................. 18
Young kids also make expressions .................................................................................................................... 18
People from New Guinea do it… ....................................................................................................................... 18
Blind people do it… ........................................................................................................................................... 19
So… ................................................................................................................................................................... 19
Example of a study ......................................................................................................................................... 20
Approach .......................................................................................................................................................... 20
Surprise! ........................................................................................................................................................... 20
One experiment ................................................................................................................................................ 20
2. Experimental contexts. ................................................................................................................................. 20
Conclusion of the experiment ........................................................................................................................... 21
Practical information about the course ............................................................................................................ 21
Overview ........................................................................................................................................................... 21
Lecture 2: What to measure for VBL ............................................................................................................... 21
Introduction ...................................................................................................................................................... 21
Terminology...................................................................................................................................................... 22
Verbal vs. non-verbalcommunication.......................................................................................................... 22
Words .......................................................................................................................................................... 22
Words, signs, morphemes ................................................................................................................................ 22
Word alternatives ........................................................................................................................................ 22
Modalities .................................................................................................................................................... 23
First modality: Speech – Source .................................................................................................................. 23
Filter............................................................................................................................................................. 23
Pitch .................................................................................................................................................................. 24
Pitch: definition ........................................................................................................................................... 24
Pitch: where do we use it for / what is its function? ................................................................................... 24
Demonstration............................................................................................................................................. 24
Another one where you can see a pitch accent: ............................................................................................... 26
Body language measures ................................................................................................................................. 27
List of body language variables ........................................................................................................................ 27
Lecture 3: Accents and prominence................................................................................................................ 27
Things we saw previously… .............................................................................................................................. 27
Highlight information structure ........................................................................................................................ 28
Information structure ....................................................................................................................................... 28
Program ............................................................................................................................................................ 29
Accents and prominence ................................................................................................................................ 29
Prominence marking ........................................................................................................................................ 29
2
, Visual cues ........................................................................................................................................................ 30
Obama video .................................................................................................................................................... 30
Link between accents and brows ...................................................................................................................... 30
Production data: newsreaders ......................................................................................................................... 30
Production data (2): beat gestures ................................................................................................................... 30
Conclusion from production data ..................................................................................................................... 31
Current lecture.................................................................................................................................................. 31
Experiment 1 .................................................................................................................................................... 31
Reaction times experiment ............................................................................................................................... 31
Task for subjects ............................................................................................................................................... 32
Congruent!!! ..................................................................................................................................................... 32
Incongruent!!! .................................................................................................................................................. 32
Results .............................................................................................................................................................. 32
Discussion Experiment 1 ................................................................................................................................... 33
Experiment 2 .................................................................................................................................................... 33
Vertical dimension (top to bottom of the face)........................................................................................... 33
Horizontal dimension: the left or right part of the face ................................................................................... 33
Original ............................................................................................................................................................. 34
Prominence detection experiment.................................................................................................................... 34
Task for subjects ............................................................................................................................................... 34
Videos ............................................................................................................................................................... 35
Results horizontal dimension (% correct) ......................................................................................................... 35
Results vertical dimension ................................................................................................................................ 35
Discussion experiment 2 ................................................................................................................................... 35
Crosslinguistic comparisons ........................................................................................................................... 36
Crosslinguistic differences in prosody ............................................................................................................... 36
Accents in different languages ......................................................................................................................... 36
English in South-Africa (Swerts and Zerbian 2010) .......................................................................................... 36
Task .................................................................................................................................................................. 37
Contrast: adjective RED cow Position: final .................................................................................................. 37
Contrast: noun (red COW) Position: final..................................................................................................... 37
Contrast: adjective (RED cow) Position: nonfinal ......................................................................................... 38
Contrast: noun (red COW) Position: nonfinal .............................................................................................. 38
Differences (based on perception results) ........................................................................................................ 38
Implications ...................................................................................................................................................... 38
Conclusion ........................................................................................................................................................ 39
Lecture 4: Manipulation of speech ................................................................................................................. 39
Overview ........................................................................................................................................................... 39
3
, Non-verbal cues ................................................................................................................................................ 39
Research ........................................................................................................................................................... 39
Artificial production ..................................................................................................................................... 39
Prominence types (see also the last lecture) .................................................................................................... 39
Prominence (meaning) types ....................................................................................................................... 40
Prominence types ............................................................................................................................................. 40
Example 1 .................................................................................................................................................... 40
Example 2 .................................................................................................................................................... 40
Example 3 .................................................................................................................................................... 41
Example 4 .................................................................................................................................................... 41
Example 5 .................................................................................................................................................... 41
Example 6 .................................................................................................................................................... 41
Example 7 .................................................................................................................................................... 41
Gives information ............................................................................................................................................. 41
Link ................................................................................................................................................................... 41
What about Marc?....................................................................................................................................... 41
Live demonstration ........................................................................................................................................... 42
Tom showed manipulating speech with the program “Praat” ......................................................................... 42
How can you say things? Speech acts, intent/valence ..................................................................................... 42
Speech acts ....................................................................................................................................................... 42
Intent/valence .................................................................................................................................................. 42
Live demonstration 2 ........................................................................................................................................ 43
Next week(s) ..................................................................................................................................................... 43
Lecture 5: Speech synthesis ............................................................................................................................ 43
Overview ........................................................................................................................................................... 43
Speech synthesis ............................................................................................................................................... 43
State of the art............................................................................................................................................. 43
Forward prediction ........................................................................................................................................... 43
Wavenet synthesis ............................................................................................................................................ 44
Naturalness .................................................................................................................................................. 44
Non-verbal cues ................................................................................................................................................ 44
Present? ....................................................................................................................................................... 44
Meaning/signaling ....................................................................................................................................... 44
More human(e)? ............................................................................................................................................... 45
Importance of affect ......................................................................................................................................... 45
Research questions in James et al. (2020) ................................................................................................... 46
Method ............................................................................................................................................................. 46
Results (1,2) ...................................................................................................................................................... 46
Results (3) ......................................................................................................................................................... 47
Schutz, Schermerhorn and Krahmer (2006) ..................................................................................................... 47
Cooperate with space robot ............................................................................................................................. 47
Method ............................................................................................................................................................. 47
Result ................................................................................................................................................................ 47
4
