Multimodal Communication and Persuasion
Inhoudsopgave
Multimodal Communication and Persuasion ............................................................................1
Hoorcollege week 1:.............................................................................................................2
Hoorcollege week 2: Gebaren en persuasie...........................................................................7
Hoorcollege week 3:........................................................................................................... 13
Hoorcollege week 4: Sense 2: Hearing, Prosody & Persuasion .............................................. 37
Hoorcollege week 5............................................................................................................ 49
Hoorcollege week 6............................................................................................................ 58
1
,Hoorcollege week 1:
Lecture 1 course intro, intro to multimodality
Today's topics
• Welcome, course intro
• What is multimodality?
• Why is multimodal communication relevant?
• Lots of examples
Teachers
The main instructors for the course are dr. Marieke hoetjes (, e4.09) and
dr. Lieke van maastricht (, e4.10).
Guest lectures will be provided by dr. Hans westerbeek, dr. Ilja croijmans, and dr. Laura speed.
Learning goals
At the end of this course, you will have achieved the following:
• You will have knowledge of and insight into the most important theories on multimodal
communication.
• You can critically evaluate research on multimodal communication, reflect on its quality,
and identify a research gap.
• You can transform the results of research on multimodal communication to (advice on)
the daily practice of persuasive communication.
• You can formulate a new research question based on the existing literature on
multimodal communication.
Course setup
The course runs for six weeks of classes.
• Lectures are on tuesdays from 15:30–17:15 in e2.53. The lecture provides an
introduction to that week’s topic.
• Seminars are on wednesdays from 15:30–17:15 in e2.53. The seminar introduces and
involves working on that week’s portfolio assignment.
Attendance at both lectures and seminars is required.
Brightspace contains the content area, which offers an overview of all weeks, including what
needs to be prepared for each meeting (readings, etc.).
Assessment
The final grade consists of two equally weighted parts:
1. 50% grade for the group portfolio. This involves weekly assignments done during and
after the seminar, in groups of around 3 students.
2. 50% grade for individual reflection meeting. This is a 15-minute meeting held in
january, discussing the course content.
More details on assessment will be provided tomorrow. Details on workload can be found in the
course syllabus under content.
Group portfolio
- inleveren voor de deadline
- Groepjes van 3
- Voorbereiden van je individuele reflectie
Individuele reflectie
2
, - Leerdoelen testen
- Kan je reflecteren
- Identificeren van onderzoek gaten
- Not: hoeveel mensen deden er mee
- Wel: waarom denk jij
◼ Welke limieten heb je kunnen identificeren.
- 14/ januari (je kan je aanmelden voor 1 dag)
Wooclap
What is multimodality?
• The term multi means "several".
• Modality refers to "one of the ways of sensation".
Multimodality involves using several ways to transfer information,
relating to reality in several ways, and processing information in
several ways. When we communicate, we typically use several
modalities.
Modalities include: image/sight/visuals, sound/hearing, smell, and
taste.
◼ Info in meer dan 1 manier die je kan verwerken als mens.
o Tekst, films, afbeeldingen. Veel hiervan is zien. Maar gesproken woorden zijn
gehoord. Soms kan je ze ruiken. Soms heb je fysiek contact.
Multimodality
Why is multimodal communication relevant?
The general idea is that multimodal communication is “better”. Furthermore, communication
is almost always multimodal. The course takes the senses as its starting point:
1. Sight, 2. Hearing, 3. Smell, 4. Taste, and 5. Touch (though not all these topics will be covered in
as much detail).
3
, The relevance of multimodality is supported by
several theories:
• Dual coding theory (paivio, 1971, 1986):
this theory suggests that you process
information both verbally and non-verbally
(or: as words and as sensory information).
◼ Presenting information in several
modalities has a positive influence on
remembering information and should
therefore facilitate communication.
• Social presence theory (short, williams & christie,
1976): this looks at the influence of digital
communication on the ‘sense of being with another’.
◼ Nonverbal communication can positively affect the
sense of closeness/social presence.
• Media richness theory (daft & lengel, 1986): this states,
“the more learning that can be pumped through a
medium, the richer the medium”.
◼ Part of this theory is the idea that a medium is
‘richer’ and more effective if it can provide several
cues simultaneously.
• In short: the idea is that multimodal communication is
“better”
• But also: communication is almost always multimodal!
This course: the senses as starting point
1. sight
2. hearing
3. smell
4. taste
5. touch
(we will not treat all these topics (in as much detail)
Let’s look at a range of examples to get a sense of what multimodality is
Sight: visuals / nonverbal communication
• Visual aspects of digital communication:
does it matter whether you use emoji or not,
and if so, which one you use?.
“Mijn zus heeft haar smartphone in de wc laten
vallen”.
“My sister dropped her smartphone in the toilet”
4
Inhoudsopgave
Multimodal Communication and Persuasion ............................................................................1
Hoorcollege week 1:.............................................................................................................2
Hoorcollege week 2: Gebaren en persuasie...........................................................................7
Hoorcollege week 3:........................................................................................................... 13
Hoorcollege week 4: Sense 2: Hearing, Prosody & Persuasion .............................................. 37
Hoorcollege week 5............................................................................................................ 49
Hoorcollege week 6............................................................................................................ 58
1
,Hoorcollege week 1:
Lecture 1 course intro, intro to multimodality
Today's topics
• Welcome, course intro
• What is multimodality?
• Why is multimodal communication relevant?
• Lots of examples
Teachers
The main instructors for the course are dr. Marieke hoetjes (, e4.09) and
dr. Lieke van maastricht (, e4.10).
Guest lectures will be provided by dr. Hans westerbeek, dr. Ilja croijmans, and dr. Laura speed.
Learning goals
At the end of this course, you will have achieved the following:
• You will have knowledge of and insight into the most important theories on multimodal
communication.
• You can critically evaluate research on multimodal communication, reflect on its quality,
and identify a research gap.
• You can transform the results of research on multimodal communication to (advice on)
the daily practice of persuasive communication.
• You can formulate a new research question based on the existing literature on
multimodal communication.
Course setup
The course runs for six weeks of classes.
• Lectures are on tuesdays from 15:30–17:15 in e2.53. The lecture provides an
introduction to that week’s topic.
• Seminars are on wednesdays from 15:30–17:15 in e2.53. The seminar introduces and
involves working on that week’s portfolio assignment.
Attendance at both lectures and seminars is required.
Brightspace contains the content area, which offers an overview of all weeks, including what
needs to be prepared for each meeting (readings, etc.).
Assessment
The final grade consists of two equally weighted parts:
1. 50% grade for the group portfolio. This involves weekly assignments done during and
after the seminar, in groups of around 3 students.
2. 50% grade for individual reflection meeting. This is a 15-minute meeting held in
january, discussing the course content.
More details on assessment will be provided tomorrow. Details on workload can be found in the
course syllabus under content.
Group portfolio
- inleveren voor de deadline
- Groepjes van 3
- Voorbereiden van je individuele reflectie
Individuele reflectie
2
, - Leerdoelen testen
- Kan je reflecteren
- Identificeren van onderzoek gaten
- Not: hoeveel mensen deden er mee
- Wel: waarom denk jij
◼ Welke limieten heb je kunnen identificeren.
- 14/ januari (je kan je aanmelden voor 1 dag)
Wooclap
What is multimodality?
• The term multi means "several".
• Modality refers to "one of the ways of sensation".
Multimodality involves using several ways to transfer information,
relating to reality in several ways, and processing information in
several ways. When we communicate, we typically use several
modalities.
Modalities include: image/sight/visuals, sound/hearing, smell, and
taste.
◼ Info in meer dan 1 manier die je kan verwerken als mens.
o Tekst, films, afbeeldingen. Veel hiervan is zien. Maar gesproken woorden zijn
gehoord. Soms kan je ze ruiken. Soms heb je fysiek contact.
Multimodality
Why is multimodal communication relevant?
The general idea is that multimodal communication is “better”. Furthermore, communication
is almost always multimodal. The course takes the senses as its starting point:
1. Sight, 2. Hearing, 3. Smell, 4. Taste, and 5. Touch (though not all these topics will be covered in
as much detail).
3
, The relevance of multimodality is supported by
several theories:
• Dual coding theory (paivio, 1971, 1986):
this theory suggests that you process
information both verbally and non-verbally
(or: as words and as sensory information).
◼ Presenting information in several
modalities has a positive influence on
remembering information and should
therefore facilitate communication.
• Social presence theory (short, williams & christie,
1976): this looks at the influence of digital
communication on the ‘sense of being with another’.
◼ Nonverbal communication can positively affect the
sense of closeness/social presence.
• Media richness theory (daft & lengel, 1986): this states,
“the more learning that can be pumped through a
medium, the richer the medium”.
◼ Part of this theory is the idea that a medium is
‘richer’ and more effective if it can provide several
cues simultaneously.
• In short: the idea is that multimodal communication is
“better”
• But also: communication is almost always multimodal!
This course: the senses as starting point
1. sight
2. hearing
3. smell
4. taste
5. touch
(we will not treat all these topics (in as much detail)
Let’s look at a range of examples to get a sense of what multimodality is
Sight: visuals / nonverbal communication
• Visual aspects of digital communication:
does it matter whether you use emoji or not,
and if so, which one you use?.
“Mijn zus heeft haar smartphone in de wc laten
vallen”.
“My sister dropped her smartphone in the toilet”
4
