Multimodal Communication summary
Article 1 A multimodal parallel architecture: A cognitive framework
for multimodal interactions
Single modality = using combinatorial structure across a sequence, such as
using sentences (with a syntactic structure) in combination with gestures or
single images (without a grammar).
Multi-modality = both the grammar of sequential words (syntax) and the
grammar of sequential images (narrative structure) as the dual packagers of
meaning.
Human communication: multimodal Speech-gesture. Does it correspond to
text–image relationships?
Visual narratives provide a challenge to multimodal communication
because the words and/or images can guide the overall meaning, and both
modalities can appear in complicated ‘‘grammatical” sequences:
sentences use a syntactic structure and sequential images use a narrative
structure. Regularly, both the grammar of sequential words and images
are used as dual packages of meaning parallel architecture is used.
Multimodal interactions: Whether a particular modality (verbal, bodily,
visual) is present, whether it uses a grammatical structure (syntax,
narrative), and whether it ‘‘dominates” the semantics of the overall
expression.
Experiment: manipulating multimodal ‘‘utterances” through diagnostic tests of
deletion (omission of elements) and substitution (replacement of elements).
Visual Narrative Grammar (VNG): (1) it is based on contemporary models of
construction grammar, not a phrase structure grammar; (2) it makes an
unambiguous separation between equal contributions of the grammar and
meaning; (3) it proposes additional modifiers (conjunction, Refiners, etc.) that
extend beyond basic canonical schemas; and (4) has been supported by
behavioral and neurocognitive research showing similarities with the
processing of syntactic structure at a sentence level.
Text-image relationships:
1. Word-Specific – Pictures illustrate but do not significantly add to the
meaning given by the text.
2. Picture-Specific – Words only provide a ‘‘soundtrack” to a visually told
sequence.
3. Duo-Specific – Both words and pictures send the same message.
4. Additive – One form amplifies or elaborates on the other.
5. Parallel – Words and images follow non-intersecting semantic discourses.
6. Interdependent – Both modalities combine to create an idea beyond the
scope of either on their own.
7. Montage – Words are treated as part of the image itself.
The parallel architecture: 3
primary components: Equal
contributions
1
, Meaning (conceptual structure) Visuals: Conceptual structure
Modality (phonology) Visuals: graphic structure
Grammar (syntactic structure) Visuals: Narrative structure
When a-conceptual
expressions non-
meaningful (because no
sense of order).
When a-tactic
expressions
Individual.
Three major categories of manifested relations between modalities: Keep in mind
that multimodal expression generally does not remain fixed to one interactive
state, but shifts between interactions.
- Autonomous: Monomodally (one modality present) fully grammatical or
atactic expressions.
- Dominant: supplementary information of one (text) to another (visuals).
The visuals could communicate the primary message without the text, but
the opposite would not be the case. Visuals should be grammatical, but the
text is not. Semantic dominance is the one with the grammar. Semantic
dominance can also be balanced = co-dominance (see picture), but one or
both need to lack grammatical structure. There can also be code-switching
(substitution)
2
, - Assertive: grammatical structure appearing in both
modalities (syntax, narrative) while the meaning
remains controlled by only one. Structure in one
modality may be lost by conveying that same
information in another modality the same basic
meaning is conveyed, but the structure changes
because of its form. When co-assertive: both the visual
and verbal modalities are necessary to form a semantic
whole greater than either of their parts. Equally
contribute its own conceptual representation to a broader composite
whole.
Lecture 1
Break apart the interacting expressions into their parts then see how those parts
are interacting.
How those parts interact. 3 ways to express concepts:
- Verbal-Auditory: Sounds. Structured sequential sounds. Spoken
language
- Verbal-Bodily: Move body. Structured sequential motions. Sign
language
- Visual-Graphic: Draw things. Structured sequential images. Visual
language
Language
Ferdinand de Saussure
• 1857-1913
• Swiss Linguist
• Began “structuralist” approach to language analysis. Structualism: when we
think about language you think about usive language (parole = language in use).
3
, Noam Chomsky
• 1928-
• American Linguist
• Shifted focus from language structure to language as a manifestation of
cognition
• Major voice in the “cognitive revolution” of the 1960s. Dealing with actual
cognition. Language is cultural.
I(nternal)- language are the properties of mind that
allow language external. Is this the same for visual
language and the rest?
E(xternal)-language: all aspects of language that
are external to the mind/brain. We see it as a
homoneous.
Langue (= structure of
the language system. Set of
words stored in peoples
mind. Different from
everyone, idoles, manifest culturally. We know that
language is the brain) and Parole (=use of
language in context, bv. Comics) are external to
mind.
Where the lines are devided, is something cultural.
Sequence of the three concepts:
Grammar: narrow version.
All three of these link up. Chomsky
would say that you just have grammar,
but that out those grammar flows the
rest (usive language).
The Parallel Architecture =
components are equally weighted.
Ray Jackendoff
• 1945-
• American Linguist, student of Noam Chomsky
• Greater integrated linguistic structure and cognition
• Also: Helped found the field of musical cognition
4
Article 1 A multimodal parallel architecture: A cognitive framework
for multimodal interactions
Single modality = using combinatorial structure across a sequence, such as
using sentences (with a syntactic structure) in combination with gestures or
single images (without a grammar).
Multi-modality = both the grammar of sequential words (syntax) and the
grammar of sequential images (narrative structure) as the dual packagers of
meaning.
Human communication: multimodal Speech-gesture. Does it correspond to
text–image relationships?
Visual narratives provide a challenge to multimodal communication
because the words and/or images can guide the overall meaning, and both
modalities can appear in complicated ‘‘grammatical” sequences:
sentences use a syntactic structure and sequential images use a narrative
structure. Regularly, both the grammar of sequential words and images
are used as dual packages of meaning parallel architecture is used.
Multimodal interactions: Whether a particular modality (verbal, bodily,
visual) is present, whether it uses a grammatical structure (syntax,
narrative), and whether it ‘‘dominates” the semantics of the overall
expression.
Experiment: manipulating multimodal ‘‘utterances” through diagnostic tests of
deletion (omission of elements) and substitution (replacement of elements).
Visual Narrative Grammar (VNG): (1) it is based on contemporary models of
construction grammar, not a phrase structure grammar; (2) it makes an
unambiguous separation between equal contributions of the grammar and
meaning; (3) it proposes additional modifiers (conjunction, Refiners, etc.) that
extend beyond basic canonical schemas; and (4) has been supported by
behavioral and neurocognitive research showing similarities with the
processing of syntactic structure at a sentence level.
Text-image relationships:
1. Word-Specific – Pictures illustrate but do not significantly add to the
meaning given by the text.
2. Picture-Specific – Words only provide a ‘‘soundtrack” to a visually told
sequence.
3. Duo-Specific – Both words and pictures send the same message.
4. Additive – One form amplifies or elaborates on the other.
5. Parallel – Words and images follow non-intersecting semantic discourses.
6. Interdependent – Both modalities combine to create an idea beyond the
scope of either on their own.
7. Montage – Words are treated as part of the image itself.
The parallel architecture: 3
primary components: Equal
contributions
1
, Meaning (conceptual structure) Visuals: Conceptual structure
Modality (phonology) Visuals: graphic structure
Grammar (syntactic structure) Visuals: Narrative structure
When a-conceptual
expressions non-
meaningful (because no
sense of order).
When a-tactic
expressions
Individual.
Three major categories of manifested relations between modalities: Keep in mind
that multimodal expression generally does not remain fixed to one interactive
state, but shifts between interactions.
- Autonomous: Monomodally (one modality present) fully grammatical or
atactic expressions.
- Dominant: supplementary information of one (text) to another (visuals).
The visuals could communicate the primary message without the text, but
the opposite would not be the case. Visuals should be grammatical, but the
text is not. Semantic dominance is the one with the grammar. Semantic
dominance can also be balanced = co-dominance (see picture), but one or
both need to lack grammatical structure. There can also be code-switching
(substitution)
2
, - Assertive: grammatical structure appearing in both
modalities (syntax, narrative) while the meaning
remains controlled by only one. Structure in one
modality may be lost by conveying that same
information in another modality the same basic
meaning is conveyed, but the structure changes
because of its form. When co-assertive: both the visual
and verbal modalities are necessary to form a semantic
whole greater than either of their parts. Equally
contribute its own conceptual representation to a broader composite
whole.
Lecture 1
Break apart the interacting expressions into their parts then see how those parts
are interacting.
How those parts interact. 3 ways to express concepts:
- Verbal-Auditory: Sounds. Structured sequential sounds. Spoken
language
- Verbal-Bodily: Move body. Structured sequential motions. Sign
language
- Visual-Graphic: Draw things. Structured sequential images. Visual
language
Language
Ferdinand de Saussure
• 1857-1913
• Swiss Linguist
• Began “structuralist” approach to language analysis. Structualism: when we
think about language you think about usive language (parole = language in use).
3
, Noam Chomsky
• 1928-
• American Linguist
• Shifted focus from language structure to language as a manifestation of
cognition
• Major voice in the “cognitive revolution” of the 1960s. Dealing with actual
cognition. Language is cultural.
I(nternal)- language are the properties of mind that
allow language external. Is this the same for visual
language and the rest?
E(xternal)-language: all aspects of language that
are external to the mind/brain. We see it as a
homoneous.
Langue (= structure of
the language system. Set of
words stored in peoples
mind. Different from
everyone, idoles, manifest culturally. We know that
language is the brain) and Parole (=use of
language in context, bv. Comics) are external to
mind.
Where the lines are devided, is something cultural.
Sequence of the three concepts:
Grammar: narrow version.
All three of these link up. Chomsky
would say that you just have grammar,
but that out those grammar flows the
rest (usive language).
The Parallel Architecture =
components are equally weighted.
Ray Jackendoff
• 1945-
• American Linguist, student of Noam Chomsky
• Greater integrated linguistic structure and cognition
• Also: Helped found the field of musical cognition
4
