Complete summary of everything combined
Assignment 1 15% 8.5
Assignment 2 25% 9.0
Total 40% 8.8
Exam 60% ?
Week 1: Strategy: linking data to business
Data analysis relates to many other elements of strategy: strategic planning,
implementation, consultancy and strategy research requires data analysis
2 phenomena why data science is important
1. The possibility of data collection in every aspect of business
2. There is huge technological development
Big data = datasets that are too large for traditional data processing systems and therefore
require new processing technologies
Big data technologies expected to be used for implementing data mining techniques,
but more often used for supporting data mining techniques
Have 3 distinct characteristics:
1. Volume = quantity of generated & stored data
2. Variety = type & nature of the data
3. Velocity = speed at which the data is generated & processed
Big data 1.0 = during web 1.0, businesses busied themselves with getting basic internet
technologies in place to they could establish web presence, build electronic capability and
improve efficiency of operations: firms are busying themselves with building capabilities to
process large data, largely in support of current operations
Big data 2.0 = Once firms have become capable of processing massive data in flexible
fashion, they begin asking what can I do now that I couldn’t do before or do better than I
could do before
Implementation of social networking component and rise of the voice of the
individual consumer
Data science = involves principles, processes, and techniques for understanding phenomena
via the analysis of data
Ultimate goal: Improving decision making
Business understanding data collection data storage data analysis implementation
We focus on data analysis
Difference data science vs. data processing
Data science = needs access to data and it often benefits from sophisticated data
engineering that data processing technologies may facilitate, but these technologies are not
data science technologies per se
Data processing = important for data-oriented business tasks that don’t involve extracting
knowledge or data-driven decision-making
,Data mining = the extraction of knowledge from data, via technologies that incorporate
these principles
Data driven decision making (DDD) = the practice of basing decisions on the analysis of
data, rather than purely on intuition
2 decisions of interest
1. Need discovery = find patterns in the data that help you understand the business/
decisions for which discoveries need to be made within data
a. E.g. Walmart after a hurricane looked at data and looked at changes in
demand after a hurricane. Saw that water was in more demand so had more
water in stock.
2. Repetitive decisions = decisions that repeat (especially at massive scale) so decision-
making can benefit from even small increases in decision-making accuracy
a. E.g. when you have a contract with telecom provider at one point you want
to switch to another provider for a better offer. If the first provider can
predict when you will switch they can retain you with a better offer.
Data driven decision making happens everywhere:
Marketing
- Online advertising (whenever you click on a link with an advert, and the page loads,
there is a bidding war going on how much people want to pay for your click)
- Recommendations for cross-selling (amazon does this when you want to buy your
photo camera, you can also buy an SD card) Things that are bought together
- Customer relationship management (Easyjet tries to give you info about how much
you travel to give you a warm feeling)
Finance
- Credit scoring and trading
- Fraud detection
- Workforce management
Retail
- Marketing (AH bonus weeks are determined by customer behavior in the store)
- Supply chain management (predict which products are going to be bank ordered and
prevent this from happening)
Data analytics = the process of examining datasets in order to draw conclusions about the
useful info they may contain
3 types of data analytics
1. Descriptive analytics (BI): What has happened?
a. Simple descriptive statistics, dashboards, charts, diagrams
b. Simple correlational methods
2. Predictive analytics: What could happen?
a. Regression, classification
b. Advanced correlation methods
3. Prescriptive analytics: What should we do?
a. A-B testing, advanced econometric techniques
b. Causality
We focus on the first 2
,Data science can help generate & sustain a CA if you align:
- Human capital
o Incentives
- Organization
o Center of excellence + local implementation (you need data scientist who can
do all the magic and local implementation with people who can speak to data
scientist and TM team)
- Culture
o Data science at core of strategy making
- Infrastructure
o No data, no DDD
Challenges in data science:
From a large mass of data, you can always find something but it’s not always 100% clear if
this is generalizable to the big crowd
Risk of over-fitting
Data analytic thinking
- Routinely transform business problems into data science problems
- Tacit skill that is only learned through trial & error
Cross Industry Standard Process for Data Mining (CRISP-DM) = model that depicts the steps
of data mining process
Also the core of the course make sure you structure your assignments according to
this model
Business understanding
Recasting the problem & designing a solution is iterative process of discovery: initial formula
may not be complete or optimal
, Data understanding
It’s important to understand strengths & limitations of the data, because rarely there is an
exact match with the problem
Data preparation phase
Is the phase in which data are manipulated and converted into forms that yield better
results (Converting data into tabular format, removing missing values, converting data to
different types)
Modeling
Output of modeling: some sort of model or pattern capturing regularities in the data
Modeling stage is primary place where data mining techniques are applied to the data
Evaluation
Purpose of evaluation stage is to assess data mining results & gain confidence that they are
valid & reliable (Also ensures that the model satisfies original business goals & Data scientist
must think of comprehensibility of model to stakeholders)
Deployment
Results of data mining are put into real use in order to realize some return on investment
From business problem to data mining problem
A sort of collaborative problem-solving between business stakeholders and data scientists
- Decomposing a business problem into (solvable) subtasks
- Matching the subtasks with known tasks for which methods and data are available
(or collectable)
- Solving the remaining non-matched subtasks (by creativity)
- Composing the subtasks to solve the overall problem
- Re-evaluation on continuous basis
Questions to answer
- What is the goal of the data science task?
- What is the business context?
- What is the data available or collectable?
- What is the appropriate method to reach the goal?
- How can the method be applied to the data?
Supervised vs. unsupervised methods
Key question: Is there a specific target variable?
- Yes: supervised learning
- No: unsupervised learning
Supervised learning = the training data has 1 feature that is the target (or dependent
variable)
- The goal is to build a model to predict the target
o Categorical: model is classification
o Numeric: model is regression
Assignment 1 15% 8.5
Assignment 2 25% 9.0
Total 40% 8.8
Exam 60% ?
Week 1: Strategy: linking data to business
Data analysis relates to many other elements of strategy: strategic planning,
implementation, consultancy and strategy research requires data analysis
2 phenomena why data science is important
1. The possibility of data collection in every aspect of business
2. There is huge technological development
Big data = datasets that are too large for traditional data processing systems and therefore
require new processing technologies
Big data technologies expected to be used for implementing data mining techniques,
but more often used for supporting data mining techniques
Have 3 distinct characteristics:
1. Volume = quantity of generated & stored data
2. Variety = type & nature of the data
3. Velocity = speed at which the data is generated & processed
Big data 1.0 = during web 1.0, businesses busied themselves with getting basic internet
technologies in place to they could establish web presence, build electronic capability and
improve efficiency of operations: firms are busying themselves with building capabilities to
process large data, largely in support of current operations
Big data 2.0 = Once firms have become capable of processing massive data in flexible
fashion, they begin asking what can I do now that I couldn’t do before or do better than I
could do before
Implementation of social networking component and rise of the voice of the
individual consumer
Data science = involves principles, processes, and techniques for understanding phenomena
via the analysis of data
Ultimate goal: Improving decision making
Business understanding data collection data storage data analysis implementation
We focus on data analysis
Difference data science vs. data processing
Data science = needs access to data and it often benefits from sophisticated data
engineering that data processing technologies may facilitate, but these technologies are not
data science technologies per se
Data processing = important for data-oriented business tasks that don’t involve extracting
knowledge or data-driven decision-making
,Data mining = the extraction of knowledge from data, via technologies that incorporate
these principles
Data driven decision making (DDD) = the practice of basing decisions on the analysis of
data, rather than purely on intuition
2 decisions of interest
1. Need discovery = find patterns in the data that help you understand the business/
decisions for which discoveries need to be made within data
a. E.g. Walmart after a hurricane looked at data and looked at changes in
demand after a hurricane. Saw that water was in more demand so had more
water in stock.
2. Repetitive decisions = decisions that repeat (especially at massive scale) so decision-
making can benefit from even small increases in decision-making accuracy
a. E.g. when you have a contract with telecom provider at one point you want
to switch to another provider for a better offer. If the first provider can
predict when you will switch they can retain you with a better offer.
Data driven decision making happens everywhere:
Marketing
- Online advertising (whenever you click on a link with an advert, and the page loads,
there is a bidding war going on how much people want to pay for your click)
- Recommendations for cross-selling (amazon does this when you want to buy your
photo camera, you can also buy an SD card) Things that are bought together
- Customer relationship management (Easyjet tries to give you info about how much
you travel to give you a warm feeling)
Finance
- Credit scoring and trading
- Fraud detection
- Workforce management
Retail
- Marketing (AH bonus weeks are determined by customer behavior in the store)
- Supply chain management (predict which products are going to be bank ordered and
prevent this from happening)
Data analytics = the process of examining datasets in order to draw conclusions about the
useful info they may contain
3 types of data analytics
1. Descriptive analytics (BI): What has happened?
a. Simple descriptive statistics, dashboards, charts, diagrams
b. Simple correlational methods
2. Predictive analytics: What could happen?
a. Regression, classification
b. Advanced correlation methods
3. Prescriptive analytics: What should we do?
a. A-B testing, advanced econometric techniques
b. Causality
We focus on the first 2
,Data science can help generate & sustain a CA if you align:
- Human capital
o Incentives
- Organization
o Center of excellence + local implementation (you need data scientist who can
do all the magic and local implementation with people who can speak to data
scientist and TM team)
- Culture
o Data science at core of strategy making
- Infrastructure
o No data, no DDD
Challenges in data science:
From a large mass of data, you can always find something but it’s not always 100% clear if
this is generalizable to the big crowd
Risk of over-fitting
Data analytic thinking
- Routinely transform business problems into data science problems
- Tacit skill that is only learned through trial & error
Cross Industry Standard Process for Data Mining (CRISP-DM) = model that depicts the steps
of data mining process
Also the core of the course make sure you structure your assignments according to
this model
Business understanding
Recasting the problem & designing a solution is iterative process of discovery: initial formula
may not be complete or optimal
, Data understanding
It’s important to understand strengths & limitations of the data, because rarely there is an
exact match with the problem
Data preparation phase
Is the phase in which data are manipulated and converted into forms that yield better
results (Converting data into tabular format, removing missing values, converting data to
different types)
Modeling
Output of modeling: some sort of model or pattern capturing regularities in the data
Modeling stage is primary place where data mining techniques are applied to the data
Evaluation
Purpose of evaluation stage is to assess data mining results & gain confidence that they are
valid & reliable (Also ensures that the model satisfies original business goals & Data scientist
must think of comprehensibility of model to stakeholders)
Deployment
Results of data mining are put into real use in order to realize some return on investment
From business problem to data mining problem
A sort of collaborative problem-solving between business stakeholders and data scientists
- Decomposing a business problem into (solvable) subtasks
- Matching the subtasks with known tasks for which methods and data are available
(or collectable)
- Solving the remaining non-matched subtasks (by creativity)
- Composing the subtasks to solve the overall problem
- Re-evaluation on continuous basis
Questions to answer
- What is the goal of the data science task?
- What is the business context?
- What is the data available or collectable?
- What is the appropriate method to reach the goal?
- How can the method be applied to the data?
Supervised vs. unsupervised methods
Key question: Is there a specific target variable?
- Yes: supervised learning
- No: unsupervised learning
Supervised learning = the training data has 1 feature that is the target (or dependent
variable)
- The goal is to build a model to predict the target
o Categorical: model is classification
o Numeric: model is regression
