NON-VERBAL COMMUNICATION
LECTURE 1: INTRODUCTION NONVERBAL COMMUNICATION
Nonverbal communication: everything that is not verbal
- Facial expression (including gaze)
- Vocal expression
- Body posture (distance)
- Gesture
- Paralinguistic signals: prosody, co-speech gesture, eyebrow accents
- Clothing, regional accents, vocabulary
Sign language is verbal
Popular statement that nonverbal features account for more than 90% of the communication, but
this is not true, think of:
- Not being able to use language
- Text messaging without emoticons
This can probably be traced back to early experimental work of Mehrabian (1971) and colleagues on
the recognition of emotions in stimuli with conflicting cues, but it is not at all clear whether it
generalizes to all aspects of non-verbal communication
People also gesture on the phone, so gestures are independent of dialogue and visibility
Culture and nonverbal communication
The expression of emotion in man and animals: research on the nonverbal communication started
with the identification of parallels between humans and animals
Universality of facial expression:
- In production
- In perception
Similarly for vocal expression of emotions
Evidence for universality, but certainly also for culture specify
Ingroup advantage: when korean and dutch listen to each other vocals > interaction effect. Dutch
listeners were a lot better on their own language.
- The study of one culture’s non-verbal behavior is the study of no culture’s non-verbal
behavior
- Most people are not weird (Western, Educated, Industrialized, Richand Democratic)
- This is often overlooked in the nonverbal domain, where universality is the default
assumption
‘Recognizing the full extent of human diversity does not mean giving up on the quest to understand
human nature. To the contrary, this recognition illuminates a journey into human nature that is more
exciting, more complex, and ultimately more consequential than has previously been suspected.’
,Examples of research:
- Gestures by politicians
- The different functions of smiles:
o Smiles in video blogs/vlogs (e.g., across the globe; alone vs. pairs)
o Sour vs. happy smiles (e.g., in quiz contexts or sports)
o Smiles by politicians during diplomatic contacts (e.g., before or after change in
relations)
- Eyebrows of newsreaders. Who flashes when?
o Swerts and Krahmer (2010) analyzed the eyebrow movements of newsreaders and
found (1) that flashes tend to co-occur with pitch accents but (2) that topic and
audience influenced frequency of flashes.
o How does this work in languages with different prosodic structures?
- Surprise:
o Ekman: one of the six basic emotions
o But: Reisenzein et al. (2006): prototypical display of surprise hardly ever occurs...
(this holds for more emotions)
o Positive vs. Negative surprise?
o Data from quiz-shows... from around the globe
- Winning and losing gestures of soccer players (male vs. female, cross-cultural, historical, ...).
- Non-verbal behavior of autistic speakers.
- Non-verbal responses to interfaces.
- Signals for turn-taking.
- Lie detection (non-verbal cues?)
Conclusion:
- Nonverbal communication is modality-independent, it can be visual-gestural, auditory, etc.
- There is evidence for both universal and culture-specific aspects of nonverbal communication
- This holds for both production and perception (hence mixed methods are essential)
- Nonverbal communication affects all forms of face-to-face interaction
- Sign language is verbal (but may integrate nonverbal elements)
,LECTURE 2: GESTURE, LANGUAGE AND CULTURE
Language is:
1. The words, their pronunciation, and the methods of combining them used and understood
by a community (Merriam-Webster)
2. The principal method of human communication, consisting of words used in a structured and
conventional way and conveyed by speech, writing, or gesture (Oxford Languages)
3. A system of communication which consists of a set of sounds and written symbols which are
used by the people of a particular country or region for talking or writing (Collins)
4. A system of communication consisting of sounds, words, and grammar (Cambridge)
Last 3 definitions: what about sign language?
1,2,4: what counts as a word (um, ah, -ed, -ish)
3: what about diaspora communities? Multilingual communities?
2: what about non-human animal communication?
Takeaway: defining language is hard
Language is something living beings to do communicate that meets the following criteria:
Arbitrary: the form doesn't resemble the meaning
Culture specific: the same meaning is expressed by different forms in different cultures
Pre-specified for form and meaning: a form reliably conveys a single meaning
Compositional: forms combine linearly according to grammatical conventions to form more
complex meaning
1. Language forms are arbitrary: the forms don’t resemble the meaning
o Example: the word write does not look or sound anything like the action of writing. The
connection between the form and meaning is arbitrary > writing and speech are language
Misinformation: gesture and signed languages are non-arbitrary. The forms resemble the
meaning > Gesture and sign are not language
2. Language forms are culture specific: a single meaning is expressed differently in different
cultures.
o Example: compare write, schrijven, écrire, ysgrifennu, ír kirjoittaa, idatzi > writing and
speech are language
Misinformation: meanings are expressed by similar forms in different cultures for both
gesture and sign → Gesture and sign are not language
3. Pre-specified for form and meaning: a form reliably conveys a single meaning
o Example: (I) Write always refers to some writing event and (ii) changing any part of
write changes the meaning (compare to bright or rite) > writing and speech are
language
Misinformation: gesture and signs can vary in form without changing the meaning >
Gesture and sign are not language
, 4. Language forms are compositional: forms combine linearly according to grammatical
conventions to form more complex meaning
o Example: "Hayden wrote on a board" ≠ "A board wrote on Hayden" ≠ "A on wrote
Hayden board" > writing and speech are language
Misinformation: gesture and signs can express complex meaning via simultaneous
combination, sometimes unexpectedly > Gesture and sign are not language
The written language bias: our conception of language is deeply influenced by a long tradition of
analyzing only written language. If Linell is right about the written language bias, then we reason
about language based on what we know about writing. The bias: Writing is definitely language, and
so "language" is made up of the things you can write down.
Writing: punctuation, strictly linear, segmented into words, words are grouped into clauses and
sentences, permanent, interpretable out of context, follows standard conventions, forms are
arbitrary
- Speech (prestige dialects) > but not permanent
- Spoken languages without writing systems/non-standard dialects > but not permanent and
don’t follow standard conventions
- Signed languages: not strictly linear, not permanent, don’t follow standard conventions and
forms are not arbitrary
- See slides
By centering the language modes furthest from the written language prototype, we can begin to see
that no language mode lives up to the written ideal (not even writing), at least not all of the time
Iconicity: connection between form and meaning
Moving past arbitrariness: the dimensionality of signing is that of life itself, and it would be stupid not
to resort to picturing, pantomiming, or pointing whenever convenient [...] But when a representation
of some four-dimensional hunk of life has to be compressed into a single dimension of speech, most
, iconicity is squeezed out. [...] Speech perforce is largely arbitrary; if we speakers take pride in that, it
is because in 50,000 years or so of talking we have learned to make a virtue of necessity
Iconicity as a semiotic resource:
- Hockett's point: language is for communicating complex information, and it makes sense for
language users to communicate as efficiently as possible
- Hands are very good at communicating about things that hands do, which is a lot!
(Manipulating objects, performing actions, directing attention)
- This makes iconicity (the resemblance of a form to its meaning) more apparent in manual
forms (signs and co-speech gestures)
But ...
- Hands aren't great for everything
- Iconic meaning occurs in every mode, just in different ways
Hands are particularly goof for conveying spatial, dimensional and motion information ironically
Primary and secondary meaning:
- In spoken languages, people often think of the visual modality as a secondary channel:
important information is conveyed in speech, bonus information in gesture
- However, gesture regularly provides information that is both necessary for interpreting the
sentence and not present in spoken mode (gesture can provide primary meaning)
- Even when gesture appear to co-express something that is expressed in speech, it is rarely
fully redundant
Flavors of iconicity:
- Speech: express is speech that resemble the meaning
o Sounds of animals: miauw, sounds of cratch/pop
o Making a words longer (this class is sooo looong)
o Repeating words (she keeps talking and talking and talking)
- Writing:
o Emphasizes words: bolding and underlying
o Spaces separating words
Multimodal ensembles:
- Primary and secondary meaning: In spoken languages, people often think of the visual
modality as a "secondary" channel: important information is conveyed in speech, bonus
information in gesture.
- However, gesture regularly provides information that is both (i) necessary for interpreting
the sentence and (ii) not present in the spoken mode (i.e. Gesture can provide primary
meaning)
- Even when gesture appears to co-express something that is expressed in speech, it is rarely
fully redundant
Wrap up:
- Language is a complicated thing to define.
- As our definition grows broader, we learn new things about more "prototypical" language
modes (e.g. identifying more iconicity in speech and writing)
- Though the hands can convey iconic meaning that cannot be expressed ironically in speech
or writing, the opposite is also true. Some meaning can't easily be expressed ironically in any
mode.
LECTURE 1: INTRODUCTION NONVERBAL COMMUNICATION
Nonverbal communication: everything that is not verbal
- Facial expression (including gaze)
- Vocal expression
- Body posture (distance)
- Gesture
- Paralinguistic signals: prosody, co-speech gesture, eyebrow accents
- Clothing, regional accents, vocabulary
Sign language is verbal
Popular statement that nonverbal features account for more than 90% of the communication, but
this is not true, think of:
- Not being able to use language
- Text messaging without emoticons
This can probably be traced back to early experimental work of Mehrabian (1971) and colleagues on
the recognition of emotions in stimuli with conflicting cues, but it is not at all clear whether it
generalizes to all aspects of non-verbal communication
People also gesture on the phone, so gestures are independent of dialogue and visibility
Culture and nonverbal communication
The expression of emotion in man and animals: research on the nonverbal communication started
with the identification of parallels between humans and animals
Universality of facial expression:
- In production
- In perception
Similarly for vocal expression of emotions
Evidence for universality, but certainly also for culture specify
Ingroup advantage: when korean and dutch listen to each other vocals > interaction effect. Dutch
listeners were a lot better on their own language.
- The study of one culture’s non-verbal behavior is the study of no culture’s non-verbal
behavior
- Most people are not weird (Western, Educated, Industrialized, Richand Democratic)
- This is often overlooked in the nonverbal domain, where universality is the default
assumption
‘Recognizing the full extent of human diversity does not mean giving up on the quest to understand
human nature. To the contrary, this recognition illuminates a journey into human nature that is more
exciting, more complex, and ultimately more consequential than has previously been suspected.’
,Examples of research:
- Gestures by politicians
- The different functions of smiles:
o Smiles in video blogs/vlogs (e.g., across the globe; alone vs. pairs)
o Sour vs. happy smiles (e.g., in quiz contexts or sports)
o Smiles by politicians during diplomatic contacts (e.g., before or after change in
relations)
- Eyebrows of newsreaders. Who flashes when?
o Swerts and Krahmer (2010) analyzed the eyebrow movements of newsreaders and
found (1) that flashes tend to co-occur with pitch accents but (2) that topic and
audience influenced frequency of flashes.
o How does this work in languages with different prosodic structures?
- Surprise:
o Ekman: one of the six basic emotions
o But: Reisenzein et al. (2006): prototypical display of surprise hardly ever occurs...
(this holds for more emotions)
o Positive vs. Negative surprise?
o Data from quiz-shows... from around the globe
- Winning and losing gestures of soccer players (male vs. female, cross-cultural, historical, ...).
- Non-verbal behavior of autistic speakers.
- Non-verbal responses to interfaces.
- Signals for turn-taking.
- Lie detection (non-verbal cues?)
Conclusion:
- Nonverbal communication is modality-independent, it can be visual-gestural, auditory, etc.
- There is evidence for both universal and culture-specific aspects of nonverbal communication
- This holds for both production and perception (hence mixed methods are essential)
- Nonverbal communication affects all forms of face-to-face interaction
- Sign language is verbal (but may integrate nonverbal elements)
,LECTURE 2: GESTURE, LANGUAGE AND CULTURE
Language is:
1. The words, their pronunciation, and the methods of combining them used and understood
by a community (Merriam-Webster)
2. The principal method of human communication, consisting of words used in a structured and
conventional way and conveyed by speech, writing, or gesture (Oxford Languages)
3. A system of communication which consists of a set of sounds and written symbols which are
used by the people of a particular country or region for talking or writing (Collins)
4. A system of communication consisting of sounds, words, and grammar (Cambridge)
Last 3 definitions: what about sign language?
1,2,4: what counts as a word (um, ah, -ed, -ish)
3: what about diaspora communities? Multilingual communities?
2: what about non-human animal communication?
Takeaway: defining language is hard
Language is something living beings to do communicate that meets the following criteria:
Arbitrary: the form doesn't resemble the meaning
Culture specific: the same meaning is expressed by different forms in different cultures
Pre-specified for form and meaning: a form reliably conveys a single meaning
Compositional: forms combine linearly according to grammatical conventions to form more
complex meaning
1. Language forms are arbitrary: the forms don’t resemble the meaning
o Example: the word write does not look or sound anything like the action of writing. The
connection between the form and meaning is arbitrary > writing and speech are language
Misinformation: gesture and signed languages are non-arbitrary. The forms resemble the
meaning > Gesture and sign are not language
2. Language forms are culture specific: a single meaning is expressed differently in different
cultures.
o Example: compare write, schrijven, écrire, ysgrifennu, ír kirjoittaa, idatzi > writing and
speech are language
Misinformation: meanings are expressed by similar forms in different cultures for both
gesture and sign → Gesture and sign are not language
3. Pre-specified for form and meaning: a form reliably conveys a single meaning
o Example: (I) Write always refers to some writing event and (ii) changing any part of
write changes the meaning (compare to bright or rite) > writing and speech are
language
Misinformation: gesture and signs can vary in form without changing the meaning >
Gesture and sign are not language
, 4. Language forms are compositional: forms combine linearly according to grammatical
conventions to form more complex meaning
o Example: "Hayden wrote on a board" ≠ "A board wrote on Hayden" ≠ "A on wrote
Hayden board" > writing and speech are language
Misinformation: gesture and signs can express complex meaning via simultaneous
combination, sometimes unexpectedly > Gesture and sign are not language
The written language bias: our conception of language is deeply influenced by a long tradition of
analyzing only written language. If Linell is right about the written language bias, then we reason
about language based on what we know about writing. The bias: Writing is definitely language, and
so "language" is made up of the things you can write down.
Writing: punctuation, strictly linear, segmented into words, words are grouped into clauses and
sentences, permanent, interpretable out of context, follows standard conventions, forms are
arbitrary
- Speech (prestige dialects) > but not permanent
- Spoken languages without writing systems/non-standard dialects > but not permanent and
don’t follow standard conventions
- Signed languages: not strictly linear, not permanent, don’t follow standard conventions and
forms are not arbitrary
- See slides
By centering the language modes furthest from the written language prototype, we can begin to see
that no language mode lives up to the written ideal (not even writing), at least not all of the time
Iconicity: connection between form and meaning
Moving past arbitrariness: the dimensionality of signing is that of life itself, and it would be stupid not
to resort to picturing, pantomiming, or pointing whenever convenient [...] But when a representation
of some four-dimensional hunk of life has to be compressed into a single dimension of speech, most
, iconicity is squeezed out. [...] Speech perforce is largely arbitrary; if we speakers take pride in that, it
is because in 50,000 years or so of talking we have learned to make a virtue of necessity
Iconicity as a semiotic resource:
- Hockett's point: language is for communicating complex information, and it makes sense for
language users to communicate as efficiently as possible
- Hands are very good at communicating about things that hands do, which is a lot!
(Manipulating objects, performing actions, directing attention)
- This makes iconicity (the resemblance of a form to its meaning) more apparent in manual
forms (signs and co-speech gestures)
But ...
- Hands aren't great for everything
- Iconic meaning occurs in every mode, just in different ways
Hands are particularly goof for conveying spatial, dimensional and motion information ironically
Primary and secondary meaning:
- In spoken languages, people often think of the visual modality as a secondary channel:
important information is conveyed in speech, bonus information in gesture
- However, gesture regularly provides information that is both necessary for interpreting the
sentence and not present in spoken mode (gesture can provide primary meaning)
- Even when gesture appear to co-express something that is expressed in speech, it is rarely
fully redundant
Flavors of iconicity:
- Speech: express is speech that resemble the meaning
o Sounds of animals: miauw, sounds of cratch/pop
o Making a words longer (this class is sooo looong)
o Repeating words (she keeps talking and talking and talking)
- Writing:
o Emphasizes words: bolding and underlying
o Spaces separating words
Multimodal ensembles:
- Primary and secondary meaning: In spoken languages, people often think of the visual
modality as a "secondary" channel: important information is conveyed in speech, bonus
information in gesture.
- However, gesture regularly provides information that is both (i) necessary for interpreting
the sentence and (ii) not present in the spoken mode (i.e. Gesture can provide primary
meaning)
- Even when gesture appears to co-express something that is expressed in speech, it is rarely
fully redundant
Wrap up:
- Language is a complicated thing to define.
- As our definition grows broader, we learn new things about more "prototypical" language
modes (e.g. identifying more iconicity in speech and writing)
- Though the hands can convey iconic meaning that cannot be expressed ironically in speech
or writing, the opposite is also true. Some meaning can't easily be expressed ironically in any
mode.
