Summary Human Media Interaction
Theory (spring)
Tilburg University 2022
880014-M-6
This document contains all relevant material for the exam, including the lectures, and the mandatory
literature of the Master course Human Media Interaction Theory (Spring)
Celeste Graumans
,Inhoud
Lecture 1; User engagement ..................................................................................................4
Lecture 1 – User engagement .............................................................................................4
Lecture 1 – Literature ..........................................................................................................9
Oh et al., 2018 – Clicking, assessing, immersing, and sharing: An empirical model of
user engagement with interactive media .........................................................................9
Important elements 1 – user engagement .........................................................................10
Lecture 2; Perception & attention..........................................................................................11
Lecture 2 – Perception & Attention....................................................................................11
Lecture 2 – Literature ........................................................................................................17
Liebherr et al., 2020 – Smartphones and attention, curse or blessing? – A review on the
effects of smartphone usage on attention, inhibition, and working memory ...................17
Important elements 2 – perception & attention ..................................................................18
Lecture 3; Attention & Cognitive load ....................................................................................18
Lecture 3 – Attention & Cognitive load ..............................................................................18
Lecture 3 – Literature ........................................................................................................24
Korbach, Brünken & Park (2017) – Measurement of cognitive load in multimedia
learning: a comparison of different objective measures .................................................24
Important elements 3 – attention & cognitive load .............................................................25
Lecture 4; Memory in the digital age .....................................................................................26
Lecture 4 – Memory ..........................................................................................................26
Lecture 4 – Literature ........................................................................................................31
Whittaker et al., 2012 – Socio-technical lifelogging: deriving design principles for a future
proof digital past ............................................................................................................31
Sparrow et al., 2011 – Google effects on memory: cognitive consequences of having
information at our fingertips ...........................................................................................32
Important elements 4 – Memory........................................................................................34
Lecture 5; Learning ..............................................................................................................35
Lecture 5 - Learning ..........................................................................................................35
Lecture 5 – Literature ........................................................................................................41
Kiili, 2005 – Digital game-based learning: Towards an experiential gaming model ........41
Important elements 5 – Learning.......................................................................................42
Lecture 6; Motivation & Emotion ...........................................................................................44
Lecture 6 – Motivation & Emotion .....................................................................................44
Lecture 6 – Literature ........................................................................................................50
Plass & Kaplan, 2016 – Emotional design in digital media for learning ..........................50
Sundar & Limperos, 2013 – Users and grats 2.0: New gratifications for new media ......52
Important elements 6 – Motivation & Emotion ...................................................................53
Lecture 7; Media multitasking ...............................................................................................55
Lecture 7 – Media multitasking .........................................................................................55
Lecture 7 – Literature ........................................................................................................60
, Uncapher & Wagner, 2018 – Minds and brains of media multitaskers: Current findings
and future directions ......................................................................................................60
Parry & le Roux, 2019 – Media multitasking and cognitive control: A systematic review of
interventions ..................................................................................................................61
Important elements 7 – Media multitasking .......................................................................61
Lecture 8; Social cognition and embodied cognition in VR....................................................63
Lecture 8 – Social cognition and embodied cognition in VR ..............................................63
Lecture 8 – Literature ........................................................................................................67
Ahn & Bailenson, 2013 – The effect of embodied experiences on self-other merging,
attitude, and helping behavior........................................................................................67
Important elements 8 – Social cognition............................................................................68
, Lecture 1; User engagement
Lecture 1 – User engagement
Engagement
- Metrics perspective: how much interaction does the user have with the product, and
for how long? (e.g., website engagement is measured by clicks/likes/comments,
sharing behavior, returning users, and time spent on the website.)
- Cognitive-psychological perspective: describing and understanding the experience
of engagement
User engagement is highly linked to user experience. There is no direct definition of the state
of mind of engagement. Some different terms and concepts refer to this state of mind. User
engagement is hard to define:
How do we call this state of mind?
All of these concepts refer to the same experience: ‘A State of mind in which all our
cognitive, affective, and perceptual-motor systems are all dedicated to the same task and the
same goal and we kind of lose track of time and surroundings while being engaged.
Why do we care about engagement?
There are quite some beneficial consequences of high user engagement
From usability to UX
- Usability: assess whether the user interface is easy to use and meets requirements
(effectiveness, efficiency, and satisfaction)
- User experience (UX): understand the deeper, personal experience of the user
(affect/emotion, fun, exciting, motivation, attitudes, attention, engagement)
Engagement is more about User Experience than usability. It is more about the experience of
users – when we talk about experiences = flow theory.
, Flow theory
It was popularized by Mihály Csikszentmihályi. He had interviews with chess players, rock
climbers, dancers and they all talked about ‘flow’ and ‘in the zone.
State of a holistic sensation that people feel when they act with total involvement, feel at their
best, perform at their best – or optimal experience of and activity engagement.
Typical flow activities
- (Extreme) sports
- Gaming
- Creative acts (composing, writing, painting, dancing)
All cognitive, affective, and physical/motor systems are all dedicated to the same task and
goal – through this you lose track of time. In a bubble – you are in a book, you can ignore
everything around you.
Challenge-Skill Balance
Both high: skills as a user and high challenge = flow
Typical flow features (9)
- Challenge-skill balance: apathy (you don’t care) – anxiety – boredom – flow
- Clear goals
- Unambiguous/immediate feedback
- Action-awareness merging (automaticity, effortlessness): you are aware of your
current action, not past actions
- Concentration (top-down attention): goal-directed attention
- Sense of control: agency
- Loss of self-consciousness (reflection): people cannot start reflecting on self-
performance, when you do you are out of the flow.
- Transformation of time: time seems to go must faster than out of the flow
experience.
- Autotelic experience (intrinsically rewarding): the task is intrinsically rewarding by
itself, no extrinsic rewards.
The ‘antecedent’ factor of an engaging experience
The ‘antecedent’ factor of an engaging experience: what
contributes to the likelihood of an optimal experience? Environment.
What are the factors that contribute?
Antecedent factors: PAT model
In computer-related activities, the likelihood of an optimal experience
depends on the interplay between person, artifact, and task.
, Person
- Trait: fixed feature of a person that doesn’t change too much over time.
- State: a more moment-to-moment feature that can change very rapidly over time.
Trait State
- Autotelic personaly (Csikszentmihalyi) - Mood
- One’s ability to experience intrinsic - Motivation
enjoyment - Arousal
- Some people are more moved by intrinsic - Focus
motivation than others
- Inherent exploratory behavior
- Personal innovativeness: willingness to try out
new technologies
- Cognitive spontaneity: individual’s playfulness
with a computer
- Trait of absorption
- (Media literacy and skills)
Artifact (new media product)
The media product itself. Artifact characteristics that contribute to the likelihood of
flow:
- Aesthetics
- Playfulness
- Vividness: breadth or number of senses (multimodality). How many
senses are being triggered?
- Speed of feedback/responsiveness: allows to main focused attention,
concentration. Critical in determining. E.g., think about the implications for
you as a user of a delay in the mouse cursor moving on the screen, or a delay in an
avatar moving after you instructed it to move. This will take you out of the flow
experience.
Task
Studying ‘task’ is challenging due to the difficulty of identifying task characteristics.
- Task characteristics: that may enhance the likelihood of flow:
o Goal-oriented (vs. exploratory) tasks: will facilitate
mostly the flow experience
o Autonomy: agency
o Variety: options to choose from
o The appropriate level of complexity: to relate to the
challenge-skill balance.
Hard to identify the task components that improve the flow experience.
PAT interactions
- Task-technology fit: task – artefact
- Clear task goals: person – task
- Sense of control over the task: person – task
- Sense of control over the artefact: person – artefact
- Perceived ease of use: person – artefact
o More important for complex tasks (person – artefact – task)
- Balance between skills and challenge: person – task
Always interact with a person or a task.
Theory (spring)
Tilburg University 2022
880014-M-6
This document contains all relevant material for the exam, including the lectures, and the mandatory
literature of the Master course Human Media Interaction Theory (Spring)
Celeste Graumans
,Inhoud
Lecture 1; User engagement ..................................................................................................4
Lecture 1 – User engagement .............................................................................................4
Lecture 1 – Literature ..........................................................................................................9
Oh et al., 2018 – Clicking, assessing, immersing, and sharing: An empirical model of
user engagement with interactive media .........................................................................9
Important elements 1 – user engagement .........................................................................10
Lecture 2; Perception & attention..........................................................................................11
Lecture 2 – Perception & Attention....................................................................................11
Lecture 2 – Literature ........................................................................................................17
Liebherr et al., 2020 – Smartphones and attention, curse or blessing? – A review on the
effects of smartphone usage on attention, inhibition, and working memory ...................17
Important elements 2 – perception & attention ..................................................................18
Lecture 3; Attention & Cognitive load ....................................................................................18
Lecture 3 – Attention & Cognitive load ..............................................................................18
Lecture 3 – Literature ........................................................................................................24
Korbach, Brünken & Park (2017) – Measurement of cognitive load in multimedia
learning: a comparison of different objective measures .................................................24
Important elements 3 – attention & cognitive load .............................................................25
Lecture 4; Memory in the digital age .....................................................................................26
Lecture 4 – Memory ..........................................................................................................26
Lecture 4 – Literature ........................................................................................................31
Whittaker et al., 2012 – Socio-technical lifelogging: deriving design principles for a future
proof digital past ............................................................................................................31
Sparrow et al., 2011 – Google effects on memory: cognitive consequences of having
information at our fingertips ...........................................................................................32
Important elements 4 – Memory........................................................................................34
Lecture 5; Learning ..............................................................................................................35
Lecture 5 - Learning ..........................................................................................................35
Lecture 5 – Literature ........................................................................................................41
Kiili, 2005 – Digital game-based learning: Towards an experiential gaming model ........41
Important elements 5 – Learning.......................................................................................42
Lecture 6; Motivation & Emotion ...........................................................................................44
Lecture 6 – Motivation & Emotion .....................................................................................44
Lecture 6 – Literature ........................................................................................................50
Plass & Kaplan, 2016 – Emotional design in digital media for learning ..........................50
Sundar & Limperos, 2013 – Users and grats 2.0: New gratifications for new media ......52
Important elements 6 – Motivation & Emotion ...................................................................53
Lecture 7; Media multitasking ...............................................................................................55
Lecture 7 – Media multitasking .........................................................................................55
Lecture 7 – Literature ........................................................................................................60
, Uncapher & Wagner, 2018 – Minds and brains of media multitaskers: Current findings
and future directions ......................................................................................................60
Parry & le Roux, 2019 – Media multitasking and cognitive control: A systematic review of
interventions ..................................................................................................................61
Important elements 7 – Media multitasking .......................................................................61
Lecture 8; Social cognition and embodied cognition in VR....................................................63
Lecture 8 – Social cognition and embodied cognition in VR ..............................................63
Lecture 8 – Literature ........................................................................................................67
Ahn & Bailenson, 2013 – The effect of embodied experiences on self-other merging,
attitude, and helping behavior........................................................................................67
Important elements 8 – Social cognition............................................................................68
, Lecture 1; User engagement
Lecture 1 – User engagement
Engagement
- Metrics perspective: how much interaction does the user have with the product, and
for how long? (e.g., website engagement is measured by clicks/likes/comments,
sharing behavior, returning users, and time spent on the website.)
- Cognitive-psychological perspective: describing and understanding the experience
of engagement
User engagement is highly linked to user experience. There is no direct definition of the state
of mind of engagement. Some different terms and concepts refer to this state of mind. User
engagement is hard to define:
How do we call this state of mind?
All of these concepts refer to the same experience: ‘A State of mind in which all our
cognitive, affective, and perceptual-motor systems are all dedicated to the same task and the
same goal and we kind of lose track of time and surroundings while being engaged.
Why do we care about engagement?
There are quite some beneficial consequences of high user engagement
From usability to UX
- Usability: assess whether the user interface is easy to use and meets requirements
(effectiveness, efficiency, and satisfaction)
- User experience (UX): understand the deeper, personal experience of the user
(affect/emotion, fun, exciting, motivation, attitudes, attention, engagement)
Engagement is more about User Experience than usability. It is more about the experience of
users – when we talk about experiences = flow theory.
, Flow theory
It was popularized by Mihály Csikszentmihályi. He had interviews with chess players, rock
climbers, dancers and they all talked about ‘flow’ and ‘in the zone.
State of a holistic sensation that people feel when they act with total involvement, feel at their
best, perform at their best – or optimal experience of and activity engagement.
Typical flow activities
- (Extreme) sports
- Gaming
- Creative acts (composing, writing, painting, dancing)
All cognitive, affective, and physical/motor systems are all dedicated to the same task and
goal – through this you lose track of time. In a bubble – you are in a book, you can ignore
everything around you.
Challenge-Skill Balance
Both high: skills as a user and high challenge = flow
Typical flow features (9)
- Challenge-skill balance: apathy (you don’t care) – anxiety – boredom – flow
- Clear goals
- Unambiguous/immediate feedback
- Action-awareness merging (automaticity, effortlessness): you are aware of your
current action, not past actions
- Concentration (top-down attention): goal-directed attention
- Sense of control: agency
- Loss of self-consciousness (reflection): people cannot start reflecting on self-
performance, when you do you are out of the flow.
- Transformation of time: time seems to go must faster than out of the flow
experience.
- Autotelic experience (intrinsically rewarding): the task is intrinsically rewarding by
itself, no extrinsic rewards.
The ‘antecedent’ factor of an engaging experience
The ‘antecedent’ factor of an engaging experience: what
contributes to the likelihood of an optimal experience? Environment.
What are the factors that contribute?
Antecedent factors: PAT model
In computer-related activities, the likelihood of an optimal experience
depends on the interplay between person, artifact, and task.
, Person
- Trait: fixed feature of a person that doesn’t change too much over time.
- State: a more moment-to-moment feature that can change very rapidly over time.
Trait State
- Autotelic personaly (Csikszentmihalyi) - Mood
- One’s ability to experience intrinsic - Motivation
enjoyment - Arousal
- Some people are more moved by intrinsic - Focus
motivation than others
- Inherent exploratory behavior
- Personal innovativeness: willingness to try out
new technologies
- Cognitive spontaneity: individual’s playfulness
with a computer
- Trait of absorption
- (Media literacy and skills)
Artifact (new media product)
The media product itself. Artifact characteristics that contribute to the likelihood of
flow:
- Aesthetics
- Playfulness
- Vividness: breadth or number of senses (multimodality). How many
senses are being triggered?
- Speed of feedback/responsiveness: allows to main focused attention,
concentration. Critical in determining. E.g., think about the implications for
you as a user of a delay in the mouse cursor moving on the screen, or a delay in an
avatar moving after you instructed it to move. This will take you out of the flow
experience.
Task
Studying ‘task’ is challenging due to the difficulty of identifying task characteristics.
- Task characteristics: that may enhance the likelihood of flow:
o Goal-oriented (vs. exploratory) tasks: will facilitate
mostly the flow experience
o Autonomy: agency
o Variety: options to choose from
o The appropriate level of complexity: to relate to the
challenge-skill balance.
Hard to identify the task components that improve the flow experience.
PAT interactions
- Task-technology fit: task – artefact
- Clear task goals: person – task
- Sense of control over the task: person – task
- Sense of control over the artefact: person – artefact
- Perceived ease of use: person – artefact
o More important for complex tasks (person – artefact – task)
- Balance between skills and challenge: person – task
Always interact with a person or a task.
