Instructor Manual, Product Analytics
Applied Data Science for Actionable Consumer Insights, by Joanne Rodrigues
Companion site for the manual: actiondatascience.com
Course Description: The goal of the course is to prepare students for real-life business
applications of product data science in industry. There are over 40 Zettabytes of consumer data
created. The sheer amount of data is staggering. How do we make sense of all of these
consumer actions? Can we influence or change consumer behavior to meet our product
directives?
Many data science courses and books only focus on the modeling component of data science.
However, in business contexts, the modeling component is often at the end of a long funnel that
starts with theory building. This course will teach you not only how to work with data, but also
how to contextual, theorize, conceptualize, and operationalize theories quantitatively to
generate actionable business insights.
The aim of the course is to provide an overview of qualitative and quantitative toolkit used in
product analytics, with an R programming component. The course takes a novel approach to
product analytics/product data science by starting with qualitative theory building techniques
and moving to quantitative techniques to validate theory propositions and test hypotheses all
within the context of the modern web or mobile product.
Course Competencies/Learning Goals:
● Understand common analytics pitfalls related to consumer behavior
● Learn how to develop theories about consumer behavior
● Understand concept building, conceptualization, and operationalization of concepts
● Understand quantitative measurement techniques of conceptual ideas
● Develop core metrics and effective KPIs for user analytics in a web product
● Understand statistical inference and the differences in correlation and causation
● Build effective A/B tests to validate hypotheses
● Build intuitive predictive models to capture user behavior in a product
● Tease out causal effects from observational data using modern, quasi-experimental
designs and statistical matching
● Improve response of marketing campaigns through uplift modeling
● Project business costs and product populations with advanced analytics techniques
● Learn how to carry out all of these quantitative techniques in the R programming
language
The remainder of this document provides chapter-by-chapter supplementary material, including:
● Learning aims
● Lists of new key terms
● Course suggestions
● Comprehension questions, with solutions
, ● Exercises, including project ideas
● Outside resources and readings
● Data Guide (at the end of the document to aid with data exercises)
● Slides for Chapters 1-6
Product Analytics Course Plan:
❖ Theory Building: Weeks 1-3
➢ The goal of the first three weeks of the course is to lay the groundwork for how to
approach product analytics questions. The focus should be on understanding
theory building techniques and sociological and psychological theory of human
behavior change.
❖ Basic Statistical Methods: Weeks 4-6
➢ The goal of the next three weeks of the course is to understand basic statistical
concepts and metric development. These three weeks can be paired with some
data cleaning exercises in R to help students better work with consumer analytics
data.
❖ Basic Predictive Methods: Weeks 7-9
➢ The goal of the next three weeks of the course is to learn predictive methods in
machine learning and demography. It’s best to pair this learning with some
traditional machine learning texts to aid students in understanding the nuts and
bolts of predictive modeling.
❖ Causal Inference Methods: Weeks 10-13
➢ The goal of the final three weeks of the course is to go over causal inference
methods. It’s best to focus on the techniques discussed in these chapters as they
are rigorous and advanced content.
Assessments: The preferred assessments for this class would be in the form of projects as they
are applied to real-life business contexts.
1. Weekly Quizzes/Homework: 40%
2. Project 1: 20%
Building a model of a web product. Observe a web product and create a theory with
hypotheses and detail how to test them. In the context of this theory, operationalize at
least two concepts and discuss your strategy and thinking.
3. Project 2: 40%
Find a natural experiment or quasi-experiment in real-life and analyze it. Write a paper
that includes the set-up, your theory and hypotheses, the mechanisms of your causal
link, and the actionable insights from this relationship. Calculate the ATE, discuss all
your assumptions, and explain the relevance of the results. Run some robustness
checks.
Note: The section “Basic, Predictive and Causal Inference Methods in R” includes R
programming examples to apply theoretical concepts and the programming exercises will be
included with the theory chapters. You can assign reading of these three chapters with the
,theory chapters—i.e., Chapter 14 during weeks 4-6 and Chapter 15 during weeks 7-9 and
Chapter 16 during weeks 10-13.
Data Resources:
Here are some open-source sites where learners can find data sets or search for data sets to
practice the skills learned in the book:
1. Kaggle Competition Data: https://www.kaggle.com/datasets
2. Census and government data: https://usa.ipums.org/usa/
3. Amazon Web Services (AWS) registry: https://registry.opendata.aws/
4. Ckan: https://ckan.org/
5. Free Code Camp: https://freeCodeCamp.org
6. Yelp open data sets: https://www.yelp.com/dataset
7. Pew research data: https://www.pewresearch.org/internet/datasets/
8. Find data: https://datasetsearch.research.google.com/
Section I: Qualitative Methods
Chapter 1: Data in Action: A Model of a Dinner Party
Learning Aims: The goal of this chapter is to change the way in which the reader approaches
product analytics.
● Understand some common pitfalls in product analytics
● Think about web products and social behavior in a new way, as a social process and not
as a problem to be solved
● Explore tools to help us understand product analytics as a social process
Key Terms: causal inference, predictive inference, outcomes, norms, problem, social process,
confirmation bias, bayes rule, counterfactual
Concepts and Themes:
● Understand that all behavior in an online product is human behavior, following the same
basic presets of other types of human behavior
● Online behavior is real and should not be aggregated away immediately, but thought of
as individual and specific human behaviors that you might observe in real life
● A common pitfall in consumer analytics is to focus on disparate aggregate facts on user
behavior without having a general theory about behavior in a web product
● There are 6 basic misconceptions about consumer behavior in a web/mobile product:
○ Social behavior is a process, not a problem to be solved
○ Social systems are open systems, meaning there are omitted or unmeasurable
variables that can affect outcomes
, ○ When exploring social behavior, there are often no clear and defined outcomes
○ Social systems are rampant with problems of incomplete or one-sided
information
○ Social systems consist of millions of potential behaviors
○ Inferring causation (or inferring why something happens) is almost impossible
Course Suggestions:
● Have the students read the beginning sections of the McKinsey reports on data
science and the future of analytics prior to the first class. See outside resources
and readings.
● One could start with this video on TED Talk, by Sociologist Stefena Broadbent,
on social processes online:
https://www.ted.com/talks/stefana_broadbent_how_the_internet_enables_intimac
y. It’s important to understand that consumer behavior on the internet is real,
social, and real-time.
● Discuss the core concepts in sociology of social norms, structures, culture, and
social behavior. Refer to the Giddens reading to build some content for class
discussion.
● Next, you could discuss the six core elements of social processes in detail,
fleshing them out with the dinner party example.
● Then, use the discussion questions to ask what additional elements might
separate a social process from a problem. Also, start using this framework to
examine other social processes.
Outside Resources and Readings:
I. Readings related to analytics in the 21st century
● Henke, Nicolaus, Bughin, Jacques, Chui, Michael, Manyika, James, Saleh, Tamin,
Wiseman, Bill, and Sethupathy, Guru. The Age of Analytics: competing in a data-driven
world. McKinsey Global Institute, 2003. https://www.mckinsey.com/business-
functions/mckinsey-analytics/our-insights/the-age-of-analytics-competing-in-a-data-
driven-world
● Manyika, James, Chui, Michael, Brown, Brad, Bughin, Jacques, Dobbs, Richard,
Roxburg, Charles and Byers, Angela. Big data: the next frontier for innovation,
competition, and productivity. McKinsey Global Institute, 2011.
https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-the-
next-frontier-for-innovation
II. Readings related to social processes
● Giddens, Anthony, Duneier, Mitchel, Appelbaum, Richard, and Carr, Deborah.
Introduction to Sociology. New York: W. W. Norton, 2009.
● https://opentextbc.ca/principlesofeconomics/chapter/16-1-the-problem-of-imperfect-
information-and-asymmetric-information/
III. Readings related to algorithms and algorithmic development
● Applegate, David, Bixby, Robert, Chvátal, Vasek, and Cook, William. The Traveling
Salesman Problem. Princeton: Princeton University Press, 2006.
● Copeland, Jack. "The Turing Test", Minds and Machines, 10, no. 4 (2000): 519-539.
Applied Data Science for Actionable Consumer Insights, by Joanne Rodrigues
Companion site for the manual: actiondatascience.com
Course Description: The goal of the course is to prepare students for real-life business
applications of product data science in industry. There are over 40 Zettabytes of consumer data
created. The sheer amount of data is staggering. How do we make sense of all of these
consumer actions? Can we influence or change consumer behavior to meet our product
directives?
Many data science courses and books only focus on the modeling component of data science.
However, in business contexts, the modeling component is often at the end of a long funnel that
starts with theory building. This course will teach you not only how to work with data, but also
how to contextual, theorize, conceptualize, and operationalize theories quantitatively to
generate actionable business insights.
The aim of the course is to provide an overview of qualitative and quantitative toolkit used in
product analytics, with an R programming component. The course takes a novel approach to
product analytics/product data science by starting with qualitative theory building techniques
and moving to quantitative techniques to validate theory propositions and test hypotheses all
within the context of the modern web or mobile product.
Course Competencies/Learning Goals:
● Understand common analytics pitfalls related to consumer behavior
● Learn how to develop theories about consumer behavior
● Understand concept building, conceptualization, and operationalization of concepts
● Understand quantitative measurement techniques of conceptual ideas
● Develop core metrics and effective KPIs for user analytics in a web product
● Understand statistical inference and the differences in correlation and causation
● Build effective A/B tests to validate hypotheses
● Build intuitive predictive models to capture user behavior in a product
● Tease out causal effects from observational data using modern, quasi-experimental
designs and statistical matching
● Improve response of marketing campaigns through uplift modeling
● Project business costs and product populations with advanced analytics techniques
● Learn how to carry out all of these quantitative techniques in the R programming
language
The remainder of this document provides chapter-by-chapter supplementary material, including:
● Learning aims
● Lists of new key terms
● Course suggestions
● Comprehension questions, with solutions
, ● Exercises, including project ideas
● Outside resources and readings
● Data Guide (at the end of the document to aid with data exercises)
● Slides for Chapters 1-6
Product Analytics Course Plan:
❖ Theory Building: Weeks 1-3
➢ The goal of the first three weeks of the course is to lay the groundwork for how to
approach product analytics questions. The focus should be on understanding
theory building techniques and sociological and psychological theory of human
behavior change.
❖ Basic Statistical Methods: Weeks 4-6
➢ The goal of the next three weeks of the course is to understand basic statistical
concepts and metric development. These three weeks can be paired with some
data cleaning exercises in R to help students better work with consumer analytics
data.
❖ Basic Predictive Methods: Weeks 7-9
➢ The goal of the next three weeks of the course is to learn predictive methods in
machine learning and demography. It’s best to pair this learning with some
traditional machine learning texts to aid students in understanding the nuts and
bolts of predictive modeling.
❖ Causal Inference Methods: Weeks 10-13
➢ The goal of the final three weeks of the course is to go over causal inference
methods. It’s best to focus on the techniques discussed in these chapters as they
are rigorous and advanced content.
Assessments: The preferred assessments for this class would be in the form of projects as they
are applied to real-life business contexts.
1. Weekly Quizzes/Homework: 40%
2. Project 1: 20%
Building a model of a web product. Observe a web product and create a theory with
hypotheses and detail how to test them. In the context of this theory, operationalize at
least two concepts and discuss your strategy and thinking.
3. Project 2: 40%
Find a natural experiment or quasi-experiment in real-life and analyze it. Write a paper
that includes the set-up, your theory and hypotheses, the mechanisms of your causal
link, and the actionable insights from this relationship. Calculate the ATE, discuss all
your assumptions, and explain the relevance of the results. Run some robustness
checks.
Note: The section “Basic, Predictive and Causal Inference Methods in R” includes R
programming examples to apply theoretical concepts and the programming exercises will be
included with the theory chapters. You can assign reading of these three chapters with the
,theory chapters—i.e., Chapter 14 during weeks 4-6 and Chapter 15 during weeks 7-9 and
Chapter 16 during weeks 10-13.
Data Resources:
Here are some open-source sites where learners can find data sets or search for data sets to
practice the skills learned in the book:
1. Kaggle Competition Data: https://www.kaggle.com/datasets
2. Census and government data: https://usa.ipums.org/usa/
3. Amazon Web Services (AWS) registry: https://registry.opendata.aws/
4. Ckan: https://ckan.org/
5. Free Code Camp: https://freeCodeCamp.org
6. Yelp open data sets: https://www.yelp.com/dataset
7. Pew research data: https://www.pewresearch.org/internet/datasets/
8. Find data: https://datasetsearch.research.google.com/
Section I: Qualitative Methods
Chapter 1: Data in Action: A Model of a Dinner Party
Learning Aims: The goal of this chapter is to change the way in which the reader approaches
product analytics.
● Understand some common pitfalls in product analytics
● Think about web products and social behavior in a new way, as a social process and not
as a problem to be solved
● Explore tools to help us understand product analytics as a social process
Key Terms: causal inference, predictive inference, outcomes, norms, problem, social process,
confirmation bias, bayes rule, counterfactual
Concepts and Themes:
● Understand that all behavior in an online product is human behavior, following the same
basic presets of other types of human behavior
● Online behavior is real and should not be aggregated away immediately, but thought of
as individual and specific human behaviors that you might observe in real life
● A common pitfall in consumer analytics is to focus on disparate aggregate facts on user
behavior without having a general theory about behavior in a web product
● There are 6 basic misconceptions about consumer behavior in a web/mobile product:
○ Social behavior is a process, not a problem to be solved
○ Social systems are open systems, meaning there are omitted or unmeasurable
variables that can affect outcomes
, ○ When exploring social behavior, there are often no clear and defined outcomes
○ Social systems are rampant with problems of incomplete or one-sided
information
○ Social systems consist of millions of potential behaviors
○ Inferring causation (or inferring why something happens) is almost impossible
Course Suggestions:
● Have the students read the beginning sections of the McKinsey reports on data
science and the future of analytics prior to the first class. See outside resources
and readings.
● One could start with this video on TED Talk, by Sociologist Stefena Broadbent,
on social processes online:
https://www.ted.com/talks/stefana_broadbent_how_the_internet_enables_intimac
y. It’s important to understand that consumer behavior on the internet is real,
social, and real-time.
● Discuss the core concepts in sociology of social norms, structures, culture, and
social behavior. Refer to the Giddens reading to build some content for class
discussion.
● Next, you could discuss the six core elements of social processes in detail,
fleshing them out with the dinner party example.
● Then, use the discussion questions to ask what additional elements might
separate a social process from a problem. Also, start using this framework to
examine other social processes.
Outside Resources and Readings:
I. Readings related to analytics in the 21st century
● Henke, Nicolaus, Bughin, Jacques, Chui, Michael, Manyika, James, Saleh, Tamin,
Wiseman, Bill, and Sethupathy, Guru. The Age of Analytics: competing in a data-driven
world. McKinsey Global Institute, 2003. https://www.mckinsey.com/business-
functions/mckinsey-analytics/our-insights/the-age-of-analytics-competing-in-a-data-
driven-world
● Manyika, James, Chui, Michael, Brown, Brad, Bughin, Jacques, Dobbs, Richard,
Roxburg, Charles and Byers, Angela. Big data: the next frontier for innovation,
competition, and productivity. McKinsey Global Institute, 2011.
https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-data-the-
next-frontier-for-innovation
II. Readings related to social processes
● Giddens, Anthony, Duneier, Mitchel, Appelbaum, Richard, and Carr, Deborah.
Introduction to Sociology. New York: W. W. Norton, 2009.
● https://opentextbc.ca/principlesofeconomics/chapter/16-1-the-problem-of-imperfect-
information-and-asymmetric-information/
III. Readings related to algorithms and algorithmic development
● Applegate, David, Bixby, Robert, Chvátal, Vasek, and Cook, William. The Traveling
Salesman Problem. Princeton: Princeton University Press, 2006.
● Copeland, Jack. "The Turing Test", Minds and Machines, 10, no. 4 (2000): 519-539.
