HUMAN MEDIA INTERACTION THEORY
Human Media Interaction Theory = the cognition-psychological study of how people interact with media and
how design features may affect these processes.
CONTENTS
Session 1: Perception & Attention ...................................................................................................................4
Information processing ........................................................................................................................................4
Attention ..............................................................................................................................................................4
Perception ............................................................................................................................................................4
Article: smartphone and attention, cure or blessing?..........................................................................................5
Important concepts session 1 ..............................................................................................................................6
Session 2: attention & cognitive load ..............................................................................................................7
Multisensory life ..................................................................................................................................................7
cognitive load & information processing .............................................................................................................7
Multisensory integration .....................................................................................................................................7
Multimodal / multisensory presentation and multimedia learning.....................................................................8
Cognitive load theory ...........................................................................................................................................9
in practise: nudging ...........................................................................................................................................11
Article: measurement of cogntiive load in multimedia learning........................................................................11
important concepts............................................................................................................................................12
Session 3: Memory in the digital age .............................................................................................................14
Memory & identity.............................................................................................................................................14
Experiencing & remembering self ......................................................................................................................14
implications for the design of new media ..........................................................................................................14
memory..............................................................................................................................................................15
Attkinson-Shiffrin theory of memory (1968) ......................................................................................................15
forgetting: how it works & how we can prevent it ............................................................................................17
Making your design Short-term and long-term memory friendly ......................................................................17
Article 1: Google effect on memory: cognitive consequences of having information at our fingertips .............18
Article 2: socio-technical lifelogging: deriving design principles for a future proof digital past ........................19
Important concepts session 3 ............................................................................................................................20
1
,Session 4: learning and digital games ............................................................................................................21
What is learning/conditioning ...........................................................................................................................21
types of learning ................................................................................................................................................21
Why is this relevant for new media design? ......................................................................................................26
Article: Digital game-based learning: towards an experiential gaming model .................................................26
important concepts session 4 ............................................................................................................................28
Session 5: emotion and motivation ...............................................................................................................30
motivation .........................................................................................................................................................30
emotion..............................................................................................................................................................33
Article 1: emotional design in digital media for learning...................................................................................37
article 2: Uses and grats 2.0: new gratifications for new media. ......................................................................41
important concepts session 5 ............................................................................................................................43
session 6: Aesthetics in human media interaction .........................................................................................46
Aesthetic experience in media design ................................................................................................................46
A workable definition.........................................................................................................................................46
Why do we like things aesthetically? .................................................................................................................47
Relation between aesthetics and usability ........................................................................................................48
relation between aesthetics and facial features ................................................................................................48
Emperical approaches: how can we study aesthetics? ......................................................................................48
How can you measure pleasure/the aesthetic experience? ..............................................................................49
General (cross-modality) principles of aesthetic pleasure .................................................................................49
Article: Assessing dimenstions of perceived visual aesthetics of web sites (lavie & tractinsky, 2004) ..............50
important concepts session 6 ............................................................................................................................52
session 7: social cognition .............................................................................................................................53
What is social cognition? ...................................................................................................................................53
How do we understand the content of other people’s minds? ..........................................................................53
Social cognition with characters in (new) media ...............................................................................................54
Entertainment-education ..................................................................................................................................54
fictional narratives can improve social cognition skills ......................................................................................55
ToM & visual cues ..............................................................................................................................................55
can non-content related design features influence social cognition in users/viewers? .....................................55
social cognition and new media ........................................................................................................................56
virtual reality can facilitate perspective taking .................................................................................................56
short summary ...................................................................................................................................................56
2
, Article: the effect of embodied experiences on self-other merging, attitude, and helping behavior, ahn &
bailenson (2013) ................................................................................................................................................56
important concepts session 7 ............................................................................................................................58
session 8: Media multitasking .......................................................................................................................60
what is a task? ...................................................................................................................................................60
Multitasking.......................................................................................................................................................60
media multitasking ............................................................................................................................................60
media multitasking & sustained effects on cognition ........................................................................................61
media multitasking & cognition ........................................................................................................................64
media multitasking questionnaire .....................................................................................................................65
ARticle: Minds and brains of media multitaskers: current findings and future directions (uncapher & wagner,
2018) ..................................................................................................................................................................66
Article: media multitasking and cognitive control: a systematic review of interventions (parry & lre roux,
2019) ..................................................................................................................................................................68
important concepts session 8 ............................................................................................................................69
session 9: user engagement ..........................................................................................................................71
Why do we care about user engagement? ........................................................................................................71
From usability to ux ...........................................................................................................................................71
flow theory.........................................................................................................................................................72
What do we study? ............................................................................................................................................73
Stages of engagement (oh et. al., 2018) ............................................................................................................75
overall knowlegde gapps ...................................................................................................................................77
ARticle: Clicking, assessing, immersing, and sharing: an empirical model of user engagement with interative
media (oh et. al., 2018)......................................................................................................................................77
important concepts session 9 ............................................................................................................................80
List of all important concepts ........................................................................................................................81
3
,SESSION 1: PERCEPTION & ATTENTION
INFORMATION PROCESSING
Based on the assumption that human processing is akin to that of computers, working
step by step (serial processing)
ATTENTION
Selective attention Refers to the ability to attend to a certain stimuli/task while disregarding others that
are irrelevant to the task at hand.
Divided attention Describes the processing/performance of two or more sources of information/tasks
at a given time.
Switching attention Refers to the ability to attend to a given task after having just attend to a different
one.
Vigilant attention Describes the process of sustaining attention to monotonous, intellectually,
unchallenging tasks over periods longer than 10 seconds up to many minutes.
Selective attention à orienting yourself to task relevant information (e.g., location, shape, color) because
there is limited capacity for processing.
Attentional control: What determines where the flashlight is pointing at?
Bottom-up attentional control Physical salience attracts attention.
Top-down attentional control Goals & expectations determine what you attend to.
Relevance history Attend to what worked for you in the past. What has been relevant for
you in the past.
PERCEPTION
From sensation to perception.
© Sensation
o Conversion of physical energy into the neural codes recognized by the brain.
o Initial step of gathering stimulus information.
4
,© Perception
o Making sense of sensation.
o Processing and organizing the raw information.
o Forming a coherent representation of the world.
o Using these representations to solve problems in the real world (navigating, comprehending,
planning)
Bottom-up perception What attentional systems have gated through.
Top-down perception © Role of pre-existing knowledge and ongoing thoughts.
© Context affects.
© Experience driven.
Perception is not a passive process; it involves active use of pre-existing knowledge (in interaction with context)
to ultimately provide meaning! Perception involves both bottom-up and top-down processes!
ARTICLE: SMARTPHONE AND ATTENTION, CURE OR BLESSING?
Smartphone impact on attention
© Focus on smartphone-related effects on cognitive functions.
© Hypothetical model assuming a differentiation between immediate & long-term effects of smartphone
use on respective cognitive functions.
© This model also describes the relations between attention, working memory, and inhibition.
© It can be assumed that smartphone usage has the potential to negatively affect cognitive functions
à but findings are limited
© It can be assumed that smartphone-related effects depend on the individual usage behavior and more
importantly vary between respective domains (selective, divided, switching and vigilant attention)
© Attentional domains relevant in the context of smartphone usage
© Findings of long-term effects are rare and somehow contrary. Based on the stated findings, it can be
solely assumed that general smartphone usage affects human’s attention.
© It can be assumed – based on dual process models – that decreased inhibitory control in smartphone
use may be the result of strong bottom-up and reduced top-down processes.
© Hypothetical model > suggest a differentiation between immediate & long-term effects of smartphone
usage.
5
, IMPORTANT CONCEPTS SESSION 1
Information The theory is based on the idea that humans process the information they receive, rather
processing than merely responding to stimuli. This perspective equates the mind to a computer,
theory which is responsible for analyzing information from the environment, the mind’s
machinery includes attention mechanisms for bringing information in, working memory
for actively manipulation information, and long-term memory for passively holding
information so that can be used in the future.
Attention The ability to focus selectively on a selected stimulus.
- Selective attention
- Divided attention
- Switching attention
- Vigilant attention
Attentional - Bottom-up attention: attentional guidance purely by externally driven factors to
control stimuli that are salient because of their inherent properties relative to the
background.
- Top-down attention: internal guidance of attention based on willful plans and
current goals.
- Relevance history: what has been relevant for you in the past.
Perception The sensory experience of the world. It involves both recognizing environmental stimuli
and actions in response to these stimuli. Through the perceptual process, we gain
information about the properties and elements of the environment that are critical to
our survival.
Perception - Bottom-up perception: Data driven and focusing on incoming sensory data
processes - Top-down perception: Information is interpreted using contextual cues and uses
previous experience and expectations.
6
Human Media Interaction Theory = the cognition-psychological study of how people interact with media and
how design features may affect these processes.
CONTENTS
Session 1: Perception & Attention ...................................................................................................................4
Information processing ........................................................................................................................................4
Attention ..............................................................................................................................................................4
Perception ............................................................................................................................................................4
Article: smartphone and attention, cure or blessing?..........................................................................................5
Important concepts session 1 ..............................................................................................................................6
Session 2: attention & cognitive load ..............................................................................................................7
Multisensory life ..................................................................................................................................................7
cognitive load & information processing .............................................................................................................7
Multisensory integration .....................................................................................................................................7
Multimodal / multisensory presentation and multimedia learning.....................................................................8
Cognitive load theory ...........................................................................................................................................9
in practise: nudging ...........................................................................................................................................11
Article: measurement of cogntiive load in multimedia learning........................................................................11
important concepts............................................................................................................................................12
Session 3: Memory in the digital age .............................................................................................................14
Memory & identity.............................................................................................................................................14
Experiencing & remembering self ......................................................................................................................14
implications for the design of new media ..........................................................................................................14
memory..............................................................................................................................................................15
Attkinson-Shiffrin theory of memory (1968) ......................................................................................................15
forgetting: how it works & how we can prevent it ............................................................................................17
Making your design Short-term and long-term memory friendly ......................................................................17
Article 1: Google effect on memory: cognitive consequences of having information at our fingertips .............18
Article 2: socio-technical lifelogging: deriving design principles for a future proof digital past ........................19
Important concepts session 3 ............................................................................................................................20
1
,Session 4: learning and digital games ............................................................................................................21
What is learning/conditioning ...........................................................................................................................21
types of learning ................................................................................................................................................21
Why is this relevant for new media design? ......................................................................................................26
Article: Digital game-based learning: towards an experiential gaming model .................................................26
important concepts session 4 ............................................................................................................................28
Session 5: emotion and motivation ...............................................................................................................30
motivation .........................................................................................................................................................30
emotion..............................................................................................................................................................33
Article 1: emotional design in digital media for learning...................................................................................37
article 2: Uses and grats 2.0: new gratifications for new media. ......................................................................41
important concepts session 5 ............................................................................................................................43
session 6: Aesthetics in human media interaction .........................................................................................46
Aesthetic experience in media design ................................................................................................................46
A workable definition.........................................................................................................................................46
Why do we like things aesthetically? .................................................................................................................47
Relation between aesthetics and usability ........................................................................................................48
relation between aesthetics and facial features ................................................................................................48
Emperical approaches: how can we study aesthetics? ......................................................................................48
How can you measure pleasure/the aesthetic experience? ..............................................................................49
General (cross-modality) principles of aesthetic pleasure .................................................................................49
Article: Assessing dimenstions of perceived visual aesthetics of web sites (lavie & tractinsky, 2004) ..............50
important concepts session 6 ............................................................................................................................52
session 7: social cognition .............................................................................................................................53
What is social cognition? ...................................................................................................................................53
How do we understand the content of other people’s minds? ..........................................................................53
Social cognition with characters in (new) media ...............................................................................................54
Entertainment-education ..................................................................................................................................54
fictional narratives can improve social cognition skills ......................................................................................55
ToM & visual cues ..............................................................................................................................................55
can non-content related design features influence social cognition in users/viewers? .....................................55
social cognition and new media ........................................................................................................................56
virtual reality can facilitate perspective taking .................................................................................................56
short summary ...................................................................................................................................................56
2
, Article: the effect of embodied experiences on self-other merging, attitude, and helping behavior, ahn &
bailenson (2013) ................................................................................................................................................56
important concepts session 7 ............................................................................................................................58
session 8: Media multitasking .......................................................................................................................60
what is a task? ...................................................................................................................................................60
Multitasking.......................................................................................................................................................60
media multitasking ............................................................................................................................................60
media multitasking & sustained effects on cognition ........................................................................................61
media multitasking & cognition ........................................................................................................................64
media multitasking questionnaire .....................................................................................................................65
ARticle: Minds and brains of media multitaskers: current findings and future directions (uncapher & wagner,
2018) ..................................................................................................................................................................66
Article: media multitasking and cognitive control: a systematic review of interventions (parry & lre roux,
2019) ..................................................................................................................................................................68
important concepts session 8 ............................................................................................................................69
session 9: user engagement ..........................................................................................................................71
Why do we care about user engagement? ........................................................................................................71
From usability to ux ...........................................................................................................................................71
flow theory.........................................................................................................................................................72
What do we study? ............................................................................................................................................73
Stages of engagement (oh et. al., 2018) ............................................................................................................75
overall knowlegde gapps ...................................................................................................................................77
ARticle: Clicking, assessing, immersing, and sharing: an empirical model of user engagement with interative
media (oh et. al., 2018)......................................................................................................................................77
important concepts session 9 ............................................................................................................................80
List of all important concepts ........................................................................................................................81
3
,SESSION 1: PERCEPTION & ATTENTION
INFORMATION PROCESSING
Based on the assumption that human processing is akin to that of computers, working
step by step (serial processing)
ATTENTION
Selective attention Refers to the ability to attend to a certain stimuli/task while disregarding others that
are irrelevant to the task at hand.
Divided attention Describes the processing/performance of two or more sources of information/tasks
at a given time.
Switching attention Refers to the ability to attend to a given task after having just attend to a different
one.
Vigilant attention Describes the process of sustaining attention to monotonous, intellectually,
unchallenging tasks over periods longer than 10 seconds up to many minutes.
Selective attention à orienting yourself to task relevant information (e.g., location, shape, color) because
there is limited capacity for processing.
Attentional control: What determines where the flashlight is pointing at?
Bottom-up attentional control Physical salience attracts attention.
Top-down attentional control Goals & expectations determine what you attend to.
Relevance history Attend to what worked for you in the past. What has been relevant for
you in the past.
PERCEPTION
From sensation to perception.
© Sensation
o Conversion of physical energy into the neural codes recognized by the brain.
o Initial step of gathering stimulus information.
4
,© Perception
o Making sense of sensation.
o Processing and organizing the raw information.
o Forming a coherent representation of the world.
o Using these representations to solve problems in the real world (navigating, comprehending,
planning)
Bottom-up perception What attentional systems have gated through.
Top-down perception © Role of pre-existing knowledge and ongoing thoughts.
© Context affects.
© Experience driven.
Perception is not a passive process; it involves active use of pre-existing knowledge (in interaction with context)
to ultimately provide meaning! Perception involves both bottom-up and top-down processes!
ARTICLE: SMARTPHONE AND ATTENTION, CURE OR BLESSING?
Smartphone impact on attention
© Focus on smartphone-related effects on cognitive functions.
© Hypothetical model assuming a differentiation between immediate & long-term effects of smartphone
use on respective cognitive functions.
© This model also describes the relations between attention, working memory, and inhibition.
© It can be assumed that smartphone usage has the potential to negatively affect cognitive functions
à but findings are limited
© It can be assumed that smartphone-related effects depend on the individual usage behavior and more
importantly vary between respective domains (selective, divided, switching and vigilant attention)
© Attentional domains relevant in the context of smartphone usage
© Findings of long-term effects are rare and somehow contrary. Based on the stated findings, it can be
solely assumed that general smartphone usage affects human’s attention.
© It can be assumed – based on dual process models – that decreased inhibitory control in smartphone
use may be the result of strong bottom-up and reduced top-down processes.
© Hypothetical model > suggest a differentiation between immediate & long-term effects of smartphone
usage.
5
, IMPORTANT CONCEPTS SESSION 1
Information The theory is based on the idea that humans process the information they receive, rather
processing than merely responding to stimuli. This perspective equates the mind to a computer,
theory which is responsible for analyzing information from the environment, the mind’s
machinery includes attention mechanisms for bringing information in, working memory
for actively manipulation information, and long-term memory for passively holding
information so that can be used in the future.
Attention The ability to focus selectively on a selected stimulus.
- Selective attention
- Divided attention
- Switching attention
- Vigilant attention
Attentional - Bottom-up attention: attentional guidance purely by externally driven factors to
control stimuli that are salient because of their inherent properties relative to the
background.
- Top-down attention: internal guidance of attention based on willful plans and
current goals.
- Relevance history: what has been relevant for you in the past.
Perception The sensory experience of the world. It involves both recognizing environmental stimuli
and actions in response to these stimuli. Through the perceptual process, we gain
information about the properties and elements of the environment that are critical to
our survival.
Perception - Bottom-up perception: Data driven and focusing on incoming sensory data
processes - Top-down perception: Information is interpreted using contextual cues and uses
previous experience and expectations.
6
