multimodal communication
Chapter 1
Multimodality is a way of understanding how we communicate using different types of signs
or methods at the same time. For example, a TV show might use speech, music, images,
and text together to tell a story. A book might use words, pictures, diagrams, and how the
page is arranged. When you're chatting in a cafeteria, you're using not just words, but also
body language, gestures, and facial expressions
Often, people use a mix of different forms of communication, like speech, writing, images,
gestures, sounds, and movement. This has become normal, especially because media and
technology have changed how we communicate both in daily life and at work. We’re
constantly using several types of media and tools at the same time.
This creates a challenge: it’s often hard to know ahead of time what forms of communication
will be used, because people combine them in so many ways. So, researchers need clear
but flexible ways to study how these combinations work.
Studying face to face interactions is something many fields look at. These situations are
made out of layers that we can look at. The difficult thing is that different experts look at
different things. a language expert looks at how they talk or use gestures, an education
expert might look at how they learn to talk about these multimodal devices etc. These fields
usually don't work together because they each have their own way of describing the
situation, this makes it hard to look at the full picture.
We need to look at all the different modes combined, not only at 1 when we look at a
situation. For example when a couple is speaking about something but then order a coffee in
between, the gestures etc. mean something else in different situations. These are different
activities and it would be confusing only analyzing the speech. Some speak about a
‘multimodal turn’= a shift in how we think about communication, especially in education and
literacy. What used to be different kinds of media: music, newspapers etc. is now ‘super
media’= Media platforms or devices that can imitate or combine the functions of multiple
traditional media, for example, a tablet that can act like a TV, newspaper, book, or music
player all in one. media convergence= The merging of different forms of media (like text,
video, audio, and images) into a single platform or device, such as a smartphone or website,
that allows you to access and interact with all of them together.
To understand and engage with multimodal content, we need to be able to analyze and think
critically about how different forms of communication are used together. So, the
“multimodal turn” means that more people, from many fields, now see the need to study
how different forms of communication work together.
1.2
Multimodal communication has been here forever, the only change is now with new media,
how science has developed over the years by breaking things into separate categories—that
has created different fields and methods for studying each kind of communication on its own.
,That has been helpful to understand each form deeply. But now we need to look at how all
these forms can work together to create shared meaning. metaphor in multimodality:
meaning multiplication= refers to how combining different forms of communication can
create a new meaning. Instead of simply adding elements together, the interaction between
modes changes how we understand each one, producing a deeper or different meaning
through their combination.
Problem: Each mode works differently so it's difficult to find the meaning when different
modes are combined. we can't list the different modes used and multiply them, just like
numbers. Each mode works in a different way so can't really ‘multiply’ them easily to find the
‘more meaning’
Another problem is that we think we can clearly identify the separate parts we want to
multiply. Multimodality as multiplication suggests that the meanings are already fixed but that
is not true because one mode can change the meaning of the other mode. In news, for
example, a person waving a flag in front of a convoy might look like they’re celebrating,
protesting, or showing courage, depending on the words and framing used with the picture.
This interdependence is hard to deal with. modes need to be interpreted in relation to
each other, not separately.
Chapter 2
Multimodality means something different in different fields and they don't all use the word
‘multimodality’ addressing multimodality in a structured way is very hard.
Firstly we need to look at the different materials and senses used. Not just physical stuff, but
materials that our senses experience in different ways. We look at senses (they are all
equal) and how language is used to create meaning.
Semiotics: study of signs and how meaning is made through them. One more key part is
society and culture, that is as important as materials and senses. Without a social and
cultural background, we wouldn’t have the basic conditions for making meaning at all. This
needs to be part of how we understand multimodal problems. Sound is a wave moving
through a medium (like air), where the material gets pushed back and forth. Sound has
typical wave features: Frequency: how often the wave repeats in a certain time. Amplitude:
how strong the wave is. Wavelength: the distance between wave peaks. Sound is a
longitudinal wave.
While sound’s physical side is fairly straightforward, how we perceive sound is much more
complex. The same applies to how we see things. It’s easy to talk about the physics of
sound, but much harder to explain what sound means to people. We don’t just hear
tones—we hear where sounds come from, what kind of thing is making them, and even
something about the space we’re in. When we use sound in language, it gets even more
complicated. Besides the physical sound, we also hear emotion, age, health, and other
qualities of the speaker—even before we understand their words. And these basic meanings
are always there, even if culture shapes how we interpret them.
Unlike visual art, which can be taken in all at once, music must be experienced over time.We
can also remember musical sequences and often predict what comes next in a song or
, composition based on patterns we've heard before. This sets a strong foundation for using
music creatively in culture. Combining music with other things like words and images is very
common like opera or lyrics with music.
Visual materials always involve multiple forms of communication.
Compositionality means that the meaning of something complex (like a sentence or image)
comes from the meanings of its smaller parts, and how those parts are put together.
mimetic/diegetic distinction: best-known differences people talk about between images and
written or spoken language is the idea that images show and language tells.
Some argue that resemblance is not something natural but something we learn through
culture. On the other side of the argument, some people say that our ability to draw or
recognize pictures isn’t totally open-ended or based only on culture.
There are limits to how we can represent things in pictures, and those limits might come
from how our brains and eyes work—not just from the culture we grow up in. If that’s true, it
means some parts of resemblance could be universal (shared by all humans).
This debate is part of a larger discussion in visual semiotics—the study of how images
communicate meaning. Many thinkers have taken different positions somewhere between
two extremes: that resemblance is totally natural, or that it's totally based on culture. This
whole topic is really important for anyone studying visual images or art. But the problem is, a
lot of past discussions don’t have solid theories or models for how different types of signs
(like images, words, gestures) actually help us create meaning.
Many people say that images can’t assert things. People say images can only show
what’s there, not what’s not there. Some think this makes images more limited than
language, since they can’t do as many things. But others argue that even language needs
context to work clearly. Today, many researchers agree we need to study visual
pragmatics—which means looking at how images are used in real-life situations—to
understand what they actually do.
Some argue that we need to move “beyond semiotics”, because traditional semiotics is
heavily based on language. we don’t need to get rid of semiotics to study visuals—we just
need to fix the parts that don’t work well for visuals.
iconography: the study of symbols, images, and visual elements and what they mean in a
particular cultural, historical, or religious context
When we move beyond just looking at “static” (non-moving) images and start thinking about
film and other moving pictures, we get even more types of meaning. In film, one newer topic
of study is how visuals affect our other senses, not just sight. This idea is called haptic
visuality. Imagine watching a movie with no sound, where someone drags their fingers
across a chalkboard or licks a rough wall—you probably still feel something, even without
sound. That’s what haptic visuality describes: visuals that make you feel things through
imagined touch.
Chapter 1
Multimodality is a way of understanding how we communicate using different types of signs
or methods at the same time. For example, a TV show might use speech, music, images,
and text together to tell a story. A book might use words, pictures, diagrams, and how the
page is arranged. When you're chatting in a cafeteria, you're using not just words, but also
body language, gestures, and facial expressions
Often, people use a mix of different forms of communication, like speech, writing, images,
gestures, sounds, and movement. This has become normal, especially because media and
technology have changed how we communicate both in daily life and at work. We’re
constantly using several types of media and tools at the same time.
This creates a challenge: it’s often hard to know ahead of time what forms of communication
will be used, because people combine them in so many ways. So, researchers need clear
but flexible ways to study how these combinations work.
Studying face to face interactions is something many fields look at. These situations are
made out of layers that we can look at. The difficult thing is that different experts look at
different things. a language expert looks at how they talk or use gestures, an education
expert might look at how they learn to talk about these multimodal devices etc. These fields
usually don't work together because they each have their own way of describing the
situation, this makes it hard to look at the full picture.
We need to look at all the different modes combined, not only at 1 when we look at a
situation. For example when a couple is speaking about something but then order a coffee in
between, the gestures etc. mean something else in different situations. These are different
activities and it would be confusing only analyzing the speech. Some speak about a
‘multimodal turn’= a shift in how we think about communication, especially in education and
literacy. What used to be different kinds of media: music, newspapers etc. is now ‘super
media’= Media platforms or devices that can imitate or combine the functions of multiple
traditional media, for example, a tablet that can act like a TV, newspaper, book, or music
player all in one. media convergence= The merging of different forms of media (like text,
video, audio, and images) into a single platform or device, such as a smartphone or website,
that allows you to access and interact with all of them together.
To understand and engage with multimodal content, we need to be able to analyze and think
critically about how different forms of communication are used together. So, the
“multimodal turn” means that more people, from many fields, now see the need to study
how different forms of communication work together.
1.2
Multimodal communication has been here forever, the only change is now with new media,
how science has developed over the years by breaking things into separate categories—that
has created different fields and methods for studying each kind of communication on its own.
,That has been helpful to understand each form deeply. But now we need to look at how all
these forms can work together to create shared meaning. metaphor in multimodality:
meaning multiplication= refers to how combining different forms of communication can
create a new meaning. Instead of simply adding elements together, the interaction between
modes changes how we understand each one, producing a deeper or different meaning
through their combination.
Problem: Each mode works differently so it's difficult to find the meaning when different
modes are combined. we can't list the different modes used and multiply them, just like
numbers. Each mode works in a different way so can't really ‘multiply’ them easily to find the
‘more meaning’
Another problem is that we think we can clearly identify the separate parts we want to
multiply. Multimodality as multiplication suggests that the meanings are already fixed but that
is not true because one mode can change the meaning of the other mode. In news, for
example, a person waving a flag in front of a convoy might look like they’re celebrating,
protesting, or showing courage, depending on the words and framing used with the picture.
This interdependence is hard to deal with. modes need to be interpreted in relation to
each other, not separately.
Chapter 2
Multimodality means something different in different fields and they don't all use the word
‘multimodality’ addressing multimodality in a structured way is very hard.
Firstly we need to look at the different materials and senses used. Not just physical stuff, but
materials that our senses experience in different ways. We look at senses (they are all
equal) and how language is used to create meaning.
Semiotics: study of signs and how meaning is made through them. One more key part is
society and culture, that is as important as materials and senses. Without a social and
cultural background, we wouldn’t have the basic conditions for making meaning at all. This
needs to be part of how we understand multimodal problems. Sound is a wave moving
through a medium (like air), where the material gets pushed back and forth. Sound has
typical wave features: Frequency: how often the wave repeats in a certain time. Amplitude:
how strong the wave is. Wavelength: the distance between wave peaks. Sound is a
longitudinal wave.
While sound’s physical side is fairly straightforward, how we perceive sound is much more
complex. The same applies to how we see things. It’s easy to talk about the physics of
sound, but much harder to explain what sound means to people. We don’t just hear
tones—we hear where sounds come from, what kind of thing is making them, and even
something about the space we’re in. When we use sound in language, it gets even more
complicated. Besides the physical sound, we also hear emotion, age, health, and other
qualities of the speaker—even before we understand their words. And these basic meanings
are always there, even if culture shapes how we interpret them.
Unlike visual art, which can be taken in all at once, music must be experienced over time.We
can also remember musical sequences and often predict what comes next in a song or
, composition based on patterns we've heard before. This sets a strong foundation for using
music creatively in culture. Combining music with other things like words and images is very
common like opera or lyrics with music.
Visual materials always involve multiple forms of communication.
Compositionality means that the meaning of something complex (like a sentence or image)
comes from the meanings of its smaller parts, and how those parts are put together.
mimetic/diegetic distinction: best-known differences people talk about between images and
written or spoken language is the idea that images show and language tells.
Some argue that resemblance is not something natural but something we learn through
culture. On the other side of the argument, some people say that our ability to draw or
recognize pictures isn’t totally open-ended or based only on culture.
There are limits to how we can represent things in pictures, and those limits might come
from how our brains and eyes work—not just from the culture we grow up in. If that’s true, it
means some parts of resemblance could be universal (shared by all humans).
This debate is part of a larger discussion in visual semiotics—the study of how images
communicate meaning. Many thinkers have taken different positions somewhere between
two extremes: that resemblance is totally natural, or that it's totally based on culture. This
whole topic is really important for anyone studying visual images or art. But the problem is, a
lot of past discussions don’t have solid theories or models for how different types of signs
(like images, words, gestures) actually help us create meaning.
Many people say that images can’t assert things. People say images can only show
what’s there, not what’s not there. Some think this makes images more limited than
language, since they can’t do as many things. But others argue that even language needs
context to work clearly. Today, many researchers agree we need to study visual
pragmatics—which means looking at how images are used in real-life situations—to
understand what they actually do.
Some argue that we need to move “beyond semiotics”, because traditional semiotics is
heavily based on language. we don’t need to get rid of semiotics to study visuals—we just
need to fix the parts that don’t work well for visuals.
iconography: the study of symbols, images, and visual elements and what they mean in a
particular cultural, historical, or religious context
When we move beyond just looking at “static” (non-moving) images and start thinking about
film and other moving pictures, we get even more types of meaning. In film, one newer topic
of study is how visuals affect our other senses, not just sight. This idea is called haptic
visuality. Imagine watching a movie with no sound, where someone drags their fingers
across a chalkboard or licks a rough wall—you probably still feel something, even without
sound. That’s what haptic visuality describes: visuals that make you feel things through
imagined touch.
