SUMMARY DATA ANALYSIS
CHAPTER 1 ~ INTRODUCTION
BIG DATA
Bit of context Get more comprehensive picture if you have integrated info (transcriptomics,
genomics, ..)
New perspectives by large scale molecular data.
Fundamentally new perspective that requires new tools --> they all fit into big data
Big data Big data= AI = deep learning
à evolution towards larger scale
Definition: is data for which conventional computer-techniques are not sufficient
anymore due to size, complexity,...
... is a disruptive trend in computer sciences. (let all our conventional ideas fall apart)
(complexity increases more compared to the evolution of computers)
Big data is characterized by 1. Volume
2. Velocity
3. Variety
4. Veracity
à is big data a reality in life sciences?
1. Volume = the amount of data we are dealing with, different than it was a long time ago
The cost of sequencing has become affordable, 2007 suddenly cost has decreased
drastically, moore’s law= this trend does not follow the getting cheaper of
sequencing, the computers cannot keep up.
1
, Moore's Law is an observation that the number of transistors in a computer chip
doubles every two years or so. As the number of transistors increases, so does
processing power. The law also states that, as the number of transistors increases,
the cost per transistor falls.
2. Velocity the speed at which it is produced, it is produces all the time (smart phone is
collecting data all the time that collects info from the environment) (data collected at
enormous speeds)
Data management gap: Speed data is produced goes faster than the growth of
people that deal with it. Impossible to have all IT people, need to be smarter how to
deal with this data
How bring data from sequencing facility to data servers in hospital --> most effective
way by bike (You have to be creative with big data)
Next step is special sequencing (single cell sequencing) à Produces massive data
Velocity is an important aspect, another trend
3. Variety in life science a lot of different data types, need to understand what you look at
heterogeneous and lots of unstructured data (eg sensor signals)
4. Veracity = data is never perfect, there might me noise, biases, info missing --> lots of aspect
that make you doubt the quality (how truthful is your data? Can you rely on it?)
It is problematic in life science, because living systems are stochastic and noisy -->
you have to deal with the messiness
Also techniques have limitations
2
,Consequences of big data à Large scale data and AI brought a new data intensive research paradigm.
(A paradigm is a standard, perspective, or set of ideas. A paradigm is a way of looking
at something.)
Large scale data they bring new paradigm à Shift in paradigm
Deep learning try to make sense of data by finding patterns in data
Data science
There is a lot of other AI than data science based AI
- Deep learning is a subset of machine learning that uses neural networks
with many layers (hence "deep") to model complex patterns in large
datasets.
- Data mining is the process of discovering patterns, correlations, and
anomalies within large sets of data using statistical and computational
techniques. (clustering, regression, discission trees, hierarchical clustering)
- Machine learning is a subset of artificial intelligence that focuses on
developing algorithms that enable computers to learn from and make
predictions or decisions based on data. (Includes supervised learning (e.g.,
linear regression, decision trees, neural networks), unsupervised learning
(e.g., k-means clustering, principal component analysis))
3
, WHAT IS DATA
What is data? Collection of data objects and their attributes
- Attribute = a property or characteristic of an object
• Examples: eye color of a person, temperature, etc.
• Attribute is also known as variable, field, characteristic, or feature
• objects= sample in lab, attribute = measure you do of the sample in the
lab
- A collection of attributes describe an object
• Object = is also known as record, point, case, sample, entity, or
instance
- Many other attributes for 1 object --> more, the better you know the object
- If have more attributes the better identify object
- Attributes can be numerical, binary, words
Example:
Attribute values Attribute values are numbers or symbols assigned to an attribute
- Distinction between attributes and attribute values
• Same attribute can be mapped to different attribute values
Example: height can be measured in cm or meters (160 cm or 1,6 m)
• Different attributes can be mapped to the same set of values
Example: Attribute values for ID and age are integers (whole numbers,
without any fractional or decimal parts)
• However properties of attribute values can still be different
Example: ID has no limit but age has a maximum and minimum value
- For each attribute type might have collection of values. E.g; post code
- Be aware what attribute value actually is
- Sometimes make choice what attribute values you allow (miles or feet)
- Can be max and min e.g. age
Attribute types • Nominal = Categories without a specific order
(based on properties) Examples: ID numbers, eye color, zip codes
• Ordinal = Categories with a meaningful order
Examples: rankings (e.g., taste of potato chips on a scale from 1-10), grades, height in
{tall, medium, short}
• Interval = Numerical data without a true zero point (The value zero represents a
complete absence of the attribute being measured)
Examples: calendar dates, temperatures in Celsius or Fahrenheit.
4
CHAPTER 1 ~ INTRODUCTION
BIG DATA
Bit of context Get more comprehensive picture if you have integrated info (transcriptomics,
genomics, ..)
New perspectives by large scale molecular data.
Fundamentally new perspective that requires new tools --> they all fit into big data
Big data Big data= AI = deep learning
à evolution towards larger scale
Definition: is data for which conventional computer-techniques are not sufficient
anymore due to size, complexity,...
... is a disruptive trend in computer sciences. (let all our conventional ideas fall apart)
(complexity increases more compared to the evolution of computers)
Big data is characterized by 1. Volume
2. Velocity
3. Variety
4. Veracity
à is big data a reality in life sciences?
1. Volume = the amount of data we are dealing with, different than it was a long time ago
The cost of sequencing has become affordable, 2007 suddenly cost has decreased
drastically, moore’s law= this trend does not follow the getting cheaper of
sequencing, the computers cannot keep up.
1
, Moore's Law is an observation that the number of transistors in a computer chip
doubles every two years or so. As the number of transistors increases, so does
processing power. The law also states that, as the number of transistors increases,
the cost per transistor falls.
2. Velocity the speed at which it is produced, it is produces all the time (smart phone is
collecting data all the time that collects info from the environment) (data collected at
enormous speeds)
Data management gap: Speed data is produced goes faster than the growth of
people that deal with it. Impossible to have all IT people, need to be smarter how to
deal with this data
How bring data from sequencing facility to data servers in hospital --> most effective
way by bike (You have to be creative with big data)
Next step is special sequencing (single cell sequencing) à Produces massive data
Velocity is an important aspect, another trend
3. Variety in life science a lot of different data types, need to understand what you look at
heterogeneous and lots of unstructured data (eg sensor signals)
4. Veracity = data is never perfect, there might me noise, biases, info missing --> lots of aspect
that make you doubt the quality (how truthful is your data? Can you rely on it?)
It is problematic in life science, because living systems are stochastic and noisy -->
you have to deal with the messiness
Also techniques have limitations
2
,Consequences of big data à Large scale data and AI brought a new data intensive research paradigm.
(A paradigm is a standard, perspective, or set of ideas. A paradigm is a way of looking
at something.)
Large scale data they bring new paradigm à Shift in paradigm
Deep learning try to make sense of data by finding patterns in data
Data science
There is a lot of other AI than data science based AI
- Deep learning is a subset of machine learning that uses neural networks
with many layers (hence "deep") to model complex patterns in large
datasets.
- Data mining is the process of discovering patterns, correlations, and
anomalies within large sets of data using statistical and computational
techniques. (clustering, regression, discission trees, hierarchical clustering)
- Machine learning is a subset of artificial intelligence that focuses on
developing algorithms that enable computers to learn from and make
predictions or decisions based on data. (Includes supervised learning (e.g.,
linear regression, decision trees, neural networks), unsupervised learning
(e.g., k-means clustering, principal component analysis))
3
, WHAT IS DATA
What is data? Collection of data objects and their attributes
- Attribute = a property or characteristic of an object
• Examples: eye color of a person, temperature, etc.
• Attribute is also known as variable, field, characteristic, or feature
• objects= sample in lab, attribute = measure you do of the sample in the
lab
- A collection of attributes describe an object
• Object = is also known as record, point, case, sample, entity, or
instance
- Many other attributes for 1 object --> more, the better you know the object
- If have more attributes the better identify object
- Attributes can be numerical, binary, words
Example:
Attribute values Attribute values are numbers or symbols assigned to an attribute
- Distinction between attributes and attribute values
• Same attribute can be mapped to different attribute values
Example: height can be measured in cm or meters (160 cm or 1,6 m)
• Different attributes can be mapped to the same set of values
Example: Attribute values for ID and age are integers (whole numbers,
without any fractional or decimal parts)
• However properties of attribute values can still be different
Example: ID has no limit but age has a maximum and minimum value
- For each attribute type might have collection of values. E.g; post code
- Be aware what attribute value actually is
- Sometimes make choice what attribute values you allow (miles or feet)
- Can be max and min e.g. age
Attribute types • Nominal = Categories without a specific order
(based on properties) Examples: ID numbers, eye color, zip codes
• Ordinal = Categories with a meaningful order
Examples: rankings (e.g., taste of potato chips on a scale from 1-10), grades, height in
{tall, medium, short}
• Interval = Numerical data without a true zero point (The value zero represents a
complete absence of the attribute being measured)
Examples: calendar dates, temperatures in Celsius or Fahrenheit.
4
