Summary Multimodal Communication
Chapter 1 Introduction: the challenge of multimodality
1.1 First steps…. a multimodal turn?
Multimodality = A way of characterizing communicative situations (considered very broadly)
which rely upon combinations of different ‘forms’ of communication to be effective.
It is not sufficient to focus on individual ‘forms of expression’ within a communicative
situation as if these forms were occurring alone. Particular forms of presentation are always
accompanied by other forms.
The multi in multimodality is about what is being combined when we talk of combinations of
diverse communicative forms.
Multimodality = A research orientation in its own right that seeks to address what happens
when diverse communicative forms combine in the service of meaning making.
Multimodality in my own words = a research field that examines how different forms of
communication work together to create meaning.
In most multimodal situations, a genuine question is: Which forms of expression are used?
Each of these different situations often uses ideas from different fields, focusing on separate
parts of what should really be seen as one whole, multimodal activity.
So a second question would be: Which disciplines are required to understand the situation?
We cannot predict the combinations of ‘modes’ which we will be confronted with. We
therefore need methods and conceptual tools that support us wherever we need to go.
The co-contextualization of forms often provides the key to their successful and effective
use.
Multimodal turn = The awareness of multimodality working its way not only into how
investigations are structured and pursued but also into our everyday ways of thinking and
acting
Media convergence = We no longer have separate media; we have media that are capable
of doing the jobs of many. Example: when an iPad shows a newspaper.
We can no longer ignore the multimodal nature of what we are looking at because media
convergence has become a normal part of how we interact with information. New forms of
communication now depend on these combinations. It is important to understand and think
critically about how different forms of expression work together.
The multimodal turn is then the willingness and, indeed, perceived need to examine
combinations of expressive resources explicitly and systematically.
,1.2 The journey ahead
Multimodality needs to be seen as always having been the norm.
Multiplication: Multimodality is more than just putting two or more modes next to each
other in what would then be an additive relationship. When things go well, more seems to
emerge from the combination than a simple sum of the parts and so a central issue for
multimodal research and practice has been just what the nature of this more might be.
Aspects are interrelated in interesting ways, playing off against each other rather than just
existing side by side multiplication
Modes = Forms of expression. A source for meaning making.
Interdependence = Difficult issue because the use of different modes together can have
influence on what each is contributing. In our analysis we need to consider what each mode
can contribute on its own. Modes presented together then need to be interpreted
With respect to one another and so cannot be considered independently.
The common assertion that the meaning in multimodality is ‘more than the sum’ of its parts
is no doubt true, but the metaphorical move from ‘sums’ to ‘multiplication’ is not quite
enough, other kinds of operations will no doubt be necessary if accounts are to be made
more precise and explanatory of actual signifying practices.
Multimodality = how different modes of communication work together to create meaning
Main message of chapter 1:
- There are many ways people combine different forms of communication to make
meaning.
- These combinations are becoming more common due to new media and
technologies.
- Studying them is complex and not straightforward, there are many challenges in
defining what exactly is being studied, how to study it, and how to make sense of it.
Many researchers start with research questions first, then go looking for evidence. But this
can be misleading, because the questions might shape the research more than the actual
nature of the multimodal material. As a result, studies might only look at parts of the
phenomenon and miss the bigger picture.
Explanation of the Figure
Label: "The pre-theoretical approach to multimodality research"
This diagram shows a circular pattern where:
Researchers begin with different questions (Research Question 1,
2, 3…).
, They then look at only certain slices of the multimodal material (like a music video, a
film, a website).
Each slice is shaped by the researcher's chosen theory or focus.
Key Point of the Figure:
The research ends up being fragmented: different researchers look at different
things, in different ways, without a shared framework.
This leads to inconsistent findings and makes it hard to relate results across studies.
Conclusion:
We need to move beyond this fragmented, question-first approach. Instead, we should
develop a more principled, systematic framework for studying multimodality. That way, we
can better understand what’s really going on in multimodal communication, not just what
our initial questions lead us to see.
Video week 1:
We rather need to become tool makers instead of tool users. We need to start at the
foundation of multimodality the book
Multimodality can be both digital and non-digital. It is a way of characterizing
communicative situations which rely upon combinations of different ‘forms’ of
communication to be effective.
Interplay of modes multimodality = extremely interdisciplinary
Each diverse situation draws potentially on different disciplinary backgrounds, picking out
facets of a situation that actually needs to be seen as a unified, multimodal activity in its own
right.
What is being combined?
Super-media = media that are able to display and simulate other media
There is a willingness and need to examine combinations of expressive resources explicitly
and systematically in and across all disciplines
Meaning combinations = multiplication
Multiple articulations = together they construct the meaning in multimodal situations
Multimodal artifact = result of using multiple semiotic modes together.
Communicative situations and media:
- What are the activities and the media involved
- Who is doing what and how
Week 2 & 3
Chapter 2. Recognizing multimodality: origins and inspirations
When working with multimodality, you need to start with navigating the problem space of
multimodal issues. This means focusing more one achieving clarity about just what
constitutes problems and challenges of multimodality: How can such problems and
challenges arise and what do they involve?
Chapter 1 Introduction: the challenge of multimodality
1.1 First steps…. a multimodal turn?
Multimodality = A way of characterizing communicative situations (considered very broadly)
which rely upon combinations of different ‘forms’ of communication to be effective.
It is not sufficient to focus on individual ‘forms of expression’ within a communicative
situation as if these forms were occurring alone. Particular forms of presentation are always
accompanied by other forms.
The multi in multimodality is about what is being combined when we talk of combinations of
diverse communicative forms.
Multimodality = A research orientation in its own right that seeks to address what happens
when diverse communicative forms combine in the service of meaning making.
Multimodality in my own words = a research field that examines how different forms of
communication work together to create meaning.
In most multimodal situations, a genuine question is: Which forms of expression are used?
Each of these different situations often uses ideas from different fields, focusing on separate
parts of what should really be seen as one whole, multimodal activity.
So a second question would be: Which disciplines are required to understand the situation?
We cannot predict the combinations of ‘modes’ which we will be confronted with. We
therefore need methods and conceptual tools that support us wherever we need to go.
The co-contextualization of forms often provides the key to their successful and effective
use.
Multimodal turn = The awareness of multimodality working its way not only into how
investigations are structured and pursued but also into our everyday ways of thinking and
acting
Media convergence = We no longer have separate media; we have media that are capable
of doing the jobs of many. Example: when an iPad shows a newspaper.
We can no longer ignore the multimodal nature of what we are looking at because media
convergence has become a normal part of how we interact with information. New forms of
communication now depend on these combinations. It is important to understand and think
critically about how different forms of expression work together.
The multimodal turn is then the willingness and, indeed, perceived need to examine
combinations of expressive resources explicitly and systematically.
,1.2 The journey ahead
Multimodality needs to be seen as always having been the norm.
Multiplication: Multimodality is more than just putting two or more modes next to each
other in what would then be an additive relationship. When things go well, more seems to
emerge from the combination than a simple sum of the parts and so a central issue for
multimodal research and practice has been just what the nature of this more might be.
Aspects are interrelated in interesting ways, playing off against each other rather than just
existing side by side multiplication
Modes = Forms of expression. A source for meaning making.
Interdependence = Difficult issue because the use of different modes together can have
influence on what each is contributing. In our analysis we need to consider what each mode
can contribute on its own. Modes presented together then need to be interpreted
With respect to one another and so cannot be considered independently.
The common assertion that the meaning in multimodality is ‘more than the sum’ of its parts
is no doubt true, but the metaphorical move from ‘sums’ to ‘multiplication’ is not quite
enough, other kinds of operations will no doubt be necessary if accounts are to be made
more precise and explanatory of actual signifying practices.
Multimodality = how different modes of communication work together to create meaning
Main message of chapter 1:
- There are many ways people combine different forms of communication to make
meaning.
- These combinations are becoming more common due to new media and
technologies.
- Studying them is complex and not straightforward, there are many challenges in
defining what exactly is being studied, how to study it, and how to make sense of it.
Many researchers start with research questions first, then go looking for evidence. But this
can be misleading, because the questions might shape the research more than the actual
nature of the multimodal material. As a result, studies might only look at parts of the
phenomenon and miss the bigger picture.
Explanation of the Figure
Label: "The pre-theoretical approach to multimodality research"
This diagram shows a circular pattern where:
Researchers begin with different questions (Research Question 1,
2, 3…).
, They then look at only certain slices of the multimodal material (like a music video, a
film, a website).
Each slice is shaped by the researcher's chosen theory or focus.
Key Point of the Figure:
The research ends up being fragmented: different researchers look at different
things, in different ways, without a shared framework.
This leads to inconsistent findings and makes it hard to relate results across studies.
Conclusion:
We need to move beyond this fragmented, question-first approach. Instead, we should
develop a more principled, systematic framework for studying multimodality. That way, we
can better understand what’s really going on in multimodal communication, not just what
our initial questions lead us to see.
Video week 1:
We rather need to become tool makers instead of tool users. We need to start at the
foundation of multimodality the book
Multimodality can be both digital and non-digital. It is a way of characterizing
communicative situations which rely upon combinations of different ‘forms’ of
communication to be effective.
Interplay of modes multimodality = extremely interdisciplinary
Each diverse situation draws potentially on different disciplinary backgrounds, picking out
facets of a situation that actually needs to be seen as a unified, multimodal activity in its own
right.
What is being combined?
Super-media = media that are able to display and simulate other media
There is a willingness and need to examine combinations of expressive resources explicitly
and systematically in and across all disciplines
Meaning combinations = multiplication
Multiple articulations = together they construct the meaning in multimodal situations
Multimodal artifact = result of using multiple semiotic modes together.
Communicative situations and media:
- What are the activities and the media involved
- Who is doing what and how
Week 2 & 3
Chapter 2. Recognizing multimodality: origins and inspirations
When working with multimodality, you need to start with navigating the problem space of
multimodal issues. This means focusing more one achieving clarity about just what
constitutes problems and challenges of multimodality: How can such problems and
challenges arise and what do they involve?
