CIS 5450 Midterm Review-Quizzes with Correct
Answers
What is the data process?
1. We start with raw data collected from the world
2. We move to clean data, in a meaningful form we can work with
3. We extract the meaningful features
4. We create models (descriptive, prescriptive models)
Why is bid data a thing now?
Computers are faster! Total volume of data has grown dramatically! Computers are cheaper
Value of data driven decision making always been high
What is big data?
Data that cannot be processed by simple means. Data that was big in 2003 might not be in 2024.
5 Vs of Big Data
1. Volume: There is a lot of data. Too much for simple tools to handle.
2. Velocity: Data arrives quickly and must be processed quickly. Working from more recent data
is a big advantage.
3. Value: Data without a model is a liability, not an asset. We need to use data to power decision
making or it is not helpful.
4. Variety: Data will come in many forms sometimes unstructured rows, columns, raw text
5. Veracity: Data can have missing pieces, or have varying levels of quality of integrity. This can
impact insights. Data can be poor and missing.
,Structured vs. Unstructured
What makes data structured is a schema. It is the kind of data a computer likes! What is a
schema?
- Structure must have any and every piece of data
- Be unambiguous
- Be computable
What is unstructured data? Text, images, videos, audio
- it is schema-less
-a human might like it but a computer won't!
Data Modalities
Modality = form, format, type
Tables: relations, spreadsheets, dataframes
Arrays: Tensors or Matrices
Document: Hierarchy, tree, XML/HTML
Tables
- Rows are facts, columns are attributes
- Columns (attributes) -- every column has a uniform type
- Every row and column pair has a value (it could be null)
- All facts have the same relation
- We can do operations on these tables effectively
- This relies on the column being homogeneous
,- We can group by some property
- Main form of data in SQL databases
Arrays
- Uniform datatype
- Rows and columns are data-dependent
- Operations: Matrix multiply, element-wise multiply, dot product, solve, compute rank
- Everything in an array is of the same type
**Key difference from tables: as you do computations with tables, the number of columns
can change depending on the schema of the table**
- With an array, all bets are off -- the number of rows impact the results
Documents
- Nested sequence of maps and lists
- Hierarchical -- a list or a map as a sequence of things
- Not as much documentation-- everything becomes a whole lot harder
What is data wrangling?
- Getting data from "raw" format into processed format
- Obviously "raw" and "processed" are context -- dependent
Often the most time consuming part!
- Our working definition: transforming data to be ready for analysis
ETL
, ETL: (general steps)
- Extract the data
- Transform (and clean) the data
-Load the data into whatever is doing your analysis
Polars vs. Pandas
Pandas can do it all, polars cannot but it is faster. It runs on multiple cores.
Core pandas functions to know
reading in data: pd.read_csv()
type checking: df.dtypes
subsetting df: df[["columns_subset"]]
change type: df["column"].astype("int")
call a function on a column: df[["column"]].apply()
lambda function: Lambda x: 1 if "true" else 0
explode column: df.explode ('col name')
aggregation: df.groupby()
built in stats functions: df[["col"].mean()df.pct_change(periods =1, axis =0)
arithmetic operations: df["col"] = df["a"]/df["b"]
Lazy vs. Eager Polars
- With the lazy API Polars doesn't run each query line-by-line but instead processes the full
query end-to-end. So polars would optimize the query.
- Lazy execution means the user needs to trigger computation in the end.
Answers
What is the data process?
1. We start with raw data collected from the world
2. We move to clean data, in a meaningful form we can work with
3. We extract the meaningful features
4. We create models (descriptive, prescriptive models)
Why is bid data a thing now?
Computers are faster! Total volume of data has grown dramatically! Computers are cheaper
Value of data driven decision making always been high
What is big data?
Data that cannot be processed by simple means. Data that was big in 2003 might not be in 2024.
5 Vs of Big Data
1. Volume: There is a lot of data. Too much for simple tools to handle.
2. Velocity: Data arrives quickly and must be processed quickly. Working from more recent data
is a big advantage.
3. Value: Data without a model is a liability, not an asset. We need to use data to power decision
making or it is not helpful.
4. Variety: Data will come in many forms sometimes unstructured rows, columns, raw text
5. Veracity: Data can have missing pieces, or have varying levels of quality of integrity. This can
impact insights. Data can be poor and missing.
,Structured vs. Unstructured
What makes data structured is a schema. It is the kind of data a computer likes! What is a
schema?
- Structure must have any and every piece of data
- Be unambiguous
- Be computable
What is unstructured data? Text, images, videos, audio
- it is schema-less
-a human might like it but a computer won't!
Data Modalities
Modality = form, format, type
Tables: relations, spreadsheets, dataframes
Arrays: Tensors or Matrices
Document: Hierarchy, tree, XML/HTML
Tables
- Rows are facts, columns are attributes
- Columns (attributes) -- every column has a uniform type
- Every row and column pair has a value (it could be null)
- All facts have the same relation
- We can do operations on these tables effectively
- This relies on the column being homogeneous
,- We can group by some property
- Main form of data in SQL databases
Arrays
- Uniform datatype
- Rows and columns are data-dependent
- Operations: Matrix multiply, element-wise multiply, dot product, solve, compute rank
- Everything in an array is of the same type
**Key difference from tables: as you do computations with tables, the number of columns
can change depending on the schema of the table**
- With an array, all bets are off -- the number of rows impact the results
Documents
- Nested sequence of maps and lists
- Hierarchical -- a list or a map as a sequence of things
- Not as much documentation-- everything becomes a whole lot harder
What is data wrangling?
- Getting data from "raw" format into processed format
- Obviously "raw" and "processed" are context -- dependent
Often the most time consuming part!
- Our working definition: transforming data to be ready for analysis
ETL
, ETL: (general steps)
- Extract the data
- Transform (and clean) the data
-Load the data into whatever is doing your analysis
Polars vs. Pandas
Pandas can do it all, polars cannot but it is faster. It runs on multiple cores.
Core pandas functions to know
reading in data: pd.read_csv()
type checking: df.dtypes
subsetting df: df[["columns_subset"]]
change type: df["column"].astype("int")
call a function on a column: df[["column"]].apply()
lambda function: Lambda x: 1 if "true" else 0
explode column: df.explode ('col name')
aggregation: df.groupby()
built in stats functions: df[["col"].mean()df.pct_change(periods =1, axis =0)
arithmetic operations: df["col"] = df["a"]/df["b"]
Lazy vs. Eager Polars
- With the lazy API Polars doesn't run each query line-by-line but instead processes the full
query end-to-end. So polars would optimize the query.
- Lazy execution means the user needs to trigger computation in the end.
