Session 1 : Introduction to
Big Data
Data ?
What is
Big Where do
challenges arise in different stages
* Moore's law >
- # of transistors on microchips doubles every two years
The Four V's
A the 4 V's :
· volume : Scale data
variety - types of data
:
·
·
velocity :
streaming speed of data
·
veracity :
imperfections in data
Volume New Working
Way of
* covers scale of data or size of datasets Traditional Computing
t
measurements : 1024'kB "TB zB CPU computation
engine of your server
>
* ↑
-
5
8yB
2
MB PB
3
GB EB * RAM ->
temporarily store data to
quickly
access information
* FaceBook : 3 billion
350
monthly active users
million photos uploaded
every day
3 shows
scale
large
↑ disk >
-
stores outcome in permanent memory
challenge & task
storing effeciently creation time
* + at same pace as * I at a
Parallel
variety Computing
* forms of data ,
traditionally structured in RDB ↑ additional cores to increase performance
* e .
. email images
g , ,
audio ,
video,
.. A more processes at the same time
Velocity
* rate of
change of data Cluster Computing HPC
* traffic spikes
a
challenges :
speed of creation increases
storing) sometimes real-time
·
-o
·
analyzing "quickly"
·
processing in parallel
transit
·
processing in
·
coping wh bursts * cluster for High Performance Computing
node
Veracity * can
only access own ram
imperfections of data * > distributed architecture
*
memory
=
* difficult to assess
quality and truthfulness of data ↑ examples for nature simulation
↳ challenge to be sure of this since data comes in a forms * for compute-intensive tasks -
> input data relatively small
volumes ,
and speed
Big Data Computing
Other Vis
* data-intensive tasks >
-
input data (very) large
A
validity -
> is data sufficient & accurate enough for use
·
data not kept in-memory
*
volatility -
> how
long is data valid ,
how
long should it be stored · few computations per data-element
data transfer
· limited runtime
by
↑ value -> what is business case for data
data
↓
locality : closer to CPU
* data partioned in smaller chunks
and divided over nodes
distributed
processing
=
, CAP-Theorem (Brewer's Theorem) Overview
* a distributed database system can
only have 213 of : + how do we
process big data
· Partition tolerance
-
: malfunction in 1 node does NOT result
in failure of entire network
the same data at the same time
·
Consistency : all nodes see
·
Availability :
system has to be
operational 100 % of the time
* Partition tolerance is not option
an
Economics of where data is stored
↳ choose between <P or CA
* on premise :
Big Data
Computing
costs -> hardware : location electricity security staffing backup gen
·
.
, , , ,
* data distributed over = nodes ,
each node : partial data >
-
off-site backup costs
for to take the
need
programming models
cloud
* new or algorithms A in
advantage of data locality
· costs -
"Pay as
you go" for object storage :
storage costs ,
data transfer
* first >
-
MapReduce costs ,
compute costs, ...
better elasticity
-
Spark >
- :
scaling upa down is
easy
>
-
backup costs (to+regions)
MapReduce
Introduction to PySpark
* developed by Google to index web
What is spark?
*
using parallel ,
distributed algorithm an a cluster
A composed of 2 procedures :
filtering
·
Map -
>
& sorting step : partition data into
smaller , independant data chunks for parallel processing
Reduce operation combines
->
summary merges or
aggregates
·
:
,
results from previous step s
*
organization
-
>
key-value pairs Spark Components
How Spark works
MapReduce to Spark
driver
*
program :
process running
the main() function & creating
the Spark Context
* SparkSession entry point toSpark
:
applications : connection to cluster
limited operations : map & reduce
* * open-source by Apache
# difficult to program * keeps benefits of MapReduce * Cluster manager : external service for acquiring resources on cluster
* requires abstraction but easier
batch Worker
*
processing *
advantages * node :
any node that can run
application code in the cluster
too much
*
reading & writing
· easier to
program New : Spark Connect
· faster data processing :
in-memory us
reading from disk
. + protocol for communication betw client application
.
& remote Spark Server
· useful for iterative algorithms
Big Data
Data ?
What is
Big Where do
challenges arise in different stages
* Moore's law >
- # of transistors on microchips doubles every two years
The Four V's
A the 4 V's :
· volume : Scale data
variety - types of data
:
·
·
velocity :
streaming speed of data
·
veracity :
imperfections in data
Volume New Working
Way of
* covers scale of data or size of datasets Traditional Computing
t
measurements : 1024'kB "TB zB CPU computation
engine of your server
>
* ↑
-
5
8yB
2
MB PB
3
GB EB * RAM ->
temporarily store data to
quickly
access information
* FaceBook : 3 billion
350
monthly active users
million photos uploaded
every day
3 shows
scale
large
↑ disk >
-
stores outcome in permanent memory
challenge & task
storing effeciently creation time
* + at same pace as * I at a
Parallel
variety Computing
* forms of data ,
traditionally structured in RDB ↑ additional cores to increase performance
* e .
. email images
g , ,
audio ,
video,
.. A more processes at the same time
Velocity
* rate of
change of data Cluster Computing HPC
* traffic spikes
a
challenges :
speed of creation increases
storing) sometimes real-time
·
-o
·
analyzing "quickly"
·
processing in parallel
transit
·
processing in
·
coping wh bursts * cluster for High Performance Computing
node
Veracity * can
only access own ram
imperfections of data * > distributed architecture
*
memory
=
* difficult to assess
quality and truthfulness of data ↑ examples for nature simulation
↳ challenge to be sure of this since data comes in a forms * for compute-intensive tasks -
> input data relatively small
volumes ,
and speed
Big Data Computing
Other Vis
* data-intensive tasks >
-
input data (very) large
A
validity -
> is data sufficient & accurate enough for use
·
data not kept in-memory
*
volatility -
> how
long is data valid ,
how
long should it be stored · few computations per data-element
data transfer
· limited runtime
by
↑ value -> what is business case for data
data
↓
locality : closer to CPU
* data partioned in smaller chunks
and divided over nodes
distributed
processing
=
, CAP-Theorem (Brewer's Theorem) Overview
* a distributed database system can
only have 213 of : + how do we
process big data
· Partition tolerance
-
: malfunction in 1 node does NOT result
in failure of entire network
the same data at the same time
·
Consistency : all nodes see
·
Availability :
system has to be
operational 100 % of the time
* Partition tolerance is not option
an
Economics of where data is stored
↳ choose between <P or CA
* on premise :
Big Data
Computing
costs -> hardware : location electricity security staffing backup gen
·
.
, , , ,
* data distributed over = nodes ,
each node : partial data >
-
off-site backup costs
for to take the
need
programming models
cloud
* new or algorithms A in
advantage of data locality
· costs -
"Pay as
you go" for object storage :
storage costs ,
data transfer
* first >
-
MapReduce costs ,
compute costs, ...
better elasticity
-
Spark >
- :
scaling upa down is
easy
>
-
backup costs (to+regions)
MapReduce
Introduction to PySpark
* developed by Google to index web
What is spark?
*
using parallel ,
distributed algorithm an a cluster
A composed of 2 procedures :
filtering
·
Map -
>
& sorting step : partition data into
smaller , independant data chunks for parallel processing
Reduce operation combines
->
summary merges or
aggregates
·
:
,
results from previous step s
*
organization
-
>
key-value pairs Spark Components
How Spark works
MapReduce to Spark
driver
*
program :
process running
the main() function & creating
the Spark Context
* SparkSession entry point toSpark
:
applications : connection to cluster
limited operations : map & reduce
* * open-source by Apache
# difficult to program * keeps benefits of MapReduce * Cluster manager : external service for acquiring resources on cluster
* requires abstraction but easier
batch Worker
*
processing *
advantages * node :
any node that can run
application code in the cluster
too much
*
reading & writing
· easier to
program New : Spark Connect
· faster data processing :
in-memory us
reading from disk
. + protocol for communication betw client application
.
& remote Spark Server
· useful for iterative algorithms
