Current Trends in CIS Research -
Lectures
Week 2: Analysing the multimodal complexity of social media (Janina Wildfeuer)
- Semiotic modes in social media postings
- How personal/professional accounts tell a story (create awareness of societal topics)
- Use of (audio)visual material
Example: Tweet Mark Rutte about Ukraine/Russia (in English)
- In the app: you can click on the photo’s
- Follow-up tweet in a thread instead of all the text in one post
Example: Instagram post about the same topic (in Dutch)
- Different pictures than on Twitter
- All text in one post
Example Facebook post Heineken:
- Picture is saying a lot
- Not much text in the post
Example video on Instagram stories (Reel)
- Short and simple video
Example TikTok
So many different communicative forms/situations cause multimodal challenges:
- Depictive media (not abstract)
- Different purposes (marketing, informing etc)
- Material from different genres, often narrative/documentary
- Various affordances (ways of using) (and constraints)
- High number of semiotic modes and interesting mode combinations
Which parts of the posting contribute to which meaning?
Which stories are told and how? How are products advertised?
“The affordances [...] are not exclusive properties of a single genre or set of genres. They are to do
with how people actually make use of them. [....] More often, the possibilities of social media genres
exaggerate the characteristics found in antecedent forms of communication.”
Page et al. (2014:18)
Methodological framework:
1. Communicative situations, media & modes (Bateman et al., 2017)
2. Interpersonal resources and subjectivity in photographs (and other artefacts) (Zappavigna,
2016)
Bateman et al. (2017: 8): “Multimodality is a way of characterising communicative situations
(considered very broadly) which rely upon combinations of different ‘forms’ of communication to be
effective”
1
, - We’re assuming that there are different modes that together express a certain meaning
- It is not accepted to only look at the words: what else is important in the communicative
form/situation?
Example: poster ‘don’t eat in lecture room’
Canvas: virtual carrier (the room where the poster hangs in) the locus of semiotic activity (where it
is located)
Sub canvas: the paper it is printed on
Genre space: informative situation in a college room
Activities, semiotic modes, what others have been doing about it, perform the actual analysis
Communicative situation: necessary conditions
Medium: higher entity: every medium makes available a certain number of semiotic modes, every
medium has his own semiotic mode
A medium is best seen as a historically stabilised site for the deployment and distribution of some
selection of semiotic modes for the achievement of varied communicative purposes
Sorting out the media and its canvases is the first key to being able to bring organisation even to
very complex situation
Bateman et al. models:
Interpretation
- Ergodic: the amount of work recipient has to do to interpret it, it is mutable ergodic if the
recipient is part of the communicative situation
- static – dynamic ((re)presentation temporality)
- 2D – 3D ((re)presentation space)
- permanent – fleeting (transience)
- observer – participant (role)
Producer has more control than consumer about the canvas: they can choose what is in there
2
, Example Instagram post Mark Rutte:
- Static
- Permanent
- Linear (image + text) (sequential)
- Ergodic (the same for everyone)/immutable (unchanging)/mutable ergodic
Canvas: Instagram post:
- Content
- Picture
- Text
- interface (like button for example)
- commentary function
it is not just about the post alone
Canvas: Instagram Story
- Asymmetric
- Dynamic
- Fleeting
- Linear
- With observers/participants
- Micro-ergodic
As communicative situations become more complex, perhaps drawing on new technological
capabilities and combinations of meaning-making strategies will prove crucial (what is done with the
material?)
Possible methods:
- user interfaces
- the role of producers
- filters
3
Lectures
Week 2: Analysing the multimodal complexity of social media (Janina Wildfeuer)
- Semiotic modes in social media postings
- How personal/professional accounts tell a story (create awareness of societal topics)
- Use of (audio)visual material
Example: Tweet Mark Rutte about Ukraine/Russia (in English)
- In the app: you can click on the photo’s
- Follow-up tweet in a thread instead of all the text in one post
Example: Instagram post about the same topic (in Dutch)
- Different pictures than on Twitter
- All text in one post
Example Facebook post Heineken:
- Picture is saying a lot
- Not much text in the post
Example video on Instagram stories (Reel)
- Short and simple video
Example TikTok
So many different communicative forms/situations cause multimodal challenges:
- Depictive media (not abstract)
- Different purposes (marketing, informing etc)
- Material from different genres, often narrative/documentary
- Various affordances (ways of using) (and constraints)
- High number of semiotic modes and interesting mode combinations
Which parts of the posting contribute to which meaning?
Which stories are told and how? How are products advertised?
“The affordances [...] are not exclusive properties of a single genre or set of genres. They are to do
with how people actually make use of them. [....] More often, the possibilities of social media genres
exaggerate the characteristics found in antecedent forms of communication.”
Page et al. (2014:18)
Methodological framework:
1. Communicative situations, media & modes (Bateman et al., 2017)
2. Interpersonal resources and subjectivity in photographs (and other artefacts) (Zappavigna,
2016)
Bateman et al. (2017: 8): “Multimodality is a way of characterising communicative situations
(considered very broadly) which rely upon combinations of different ‘forms’ of communication to be
effective”
1
, - We’re assuming that there are different modes that together express a certain meaning
- It is not accepted to only look at the words: what else is important in the communicative
form/situation?
Example: poster ‘don’t eat in lecture room’
Canvas: virtual carrier (the room where the poster hangs in) the locus of semiotic activity (where it
is located)
Sub canvas: the paper it is printed on
Genre space: informative situation in a college room
Activities, semiotic modes, what others have been doing about it, perform the actual analysis
Communicative situation: necessary conditions
Medium: higher entity: every medium makes available a certain number of semiotic modes, every
medium has his own semiotic mode
A medium is best seen as a historically stabilised site for the deployment and distribution of some
selection of semiotic modes for the achievement of varied communicative purposes
Sorting out the media and its canvases is the first key to being able to bring organisation even to
very complex situation
Bateman et al. models:
Interpretation
- Ergodic: the amount of work recipient has to do to interpret it, it is mutable ergodic if the
recipient is part of the communicative situation
- static – dynamic ((re)presentation temporality)
- 2D – 3D ((re)presentation space)
- permanent – fleeting (transience)
- observer – participant (role)
Producer has more control than consumer about the canvas: they can choose what is in there
2
, Example Instagram post Mark Rutte:
- Static
- Permanent
- Linear (image + text) (sequential)
- Ergodic (the same for everyone)/immutable (unchanging)/mutable ergodic
Canvas: Instagram post:
- Content
- Picture
- Text
- interface (like button for example)
- commentary function
it is not just about the post alone
Canvas: Instagram Story
- Asymmetric
- Dynamic
- Fleeting
- Linear
- With observers/participants
- Micro-ergodic
As communicative situations become more complex, perhaps drawing on new technological
capabilities and combinations of meaning-making strategies will prove crucial (what is done with the
material?)
Possible methods:
- user interfaces
- the role of producers
- filters
3
