BUAN 6346 BIG DATA STUDY GUIDE
Partitioning (data storage) - Answers - Subdividing table storage into HDFS
subdirectories by column value to avoid full scans.
Full table scan - Answers - Default non-partitioned behavior where every file in the
directory is read during analysis.
Partition column - Answers - A virtual column whose value is encoded in the directory
name, not in the data file.
Partition directory naming format - Answers - column=value format, e.g., state=CA or
call_date=2014-10-02.
Dynamic partitioning - Answers - Automatic creation of partitions at load time based on
the last column in SELECT/INSERT.
Static partitioning - Answers - Manually specifying the target partition using ALTER
TABLE ... ADD PARTITION.
hive.exec.dynamic.partition - Answers - Hive property that must be set to true to
enable dynamic partitioning.
hive.exec.dynamic.partition.mode - Answers - Hive property set to nonstrict to allow all
partitions in a query to be dynamic.
hive.exec.max.dynamic.partitions.pernode - Answers - Hive limit on dynamic partitions
created per node in a query; default is 100.
hive.exec.max.dynamic.partitions - Answers - Hive limit on total dynamic partitions
created per HiveQL statement; default is 1000.
hive.exec.max.created.files - Answers - Hive limit on total files created across all
nodes by a query; default is 100000.
MSCK REPAIR TABLE - Answers - Hive command that recovers partition metadata
created outside Hive (e.g., via Spark or MapReduce).
LOAD DATA INPATH ... PARTITION(...) - Answers - Hive command that moves an
HDFS file into the specified partition subdirectory.
Hive-only LOAD shortcut - Answers - Hive automatically creates non-existent partition
metadata and subdirectories on LOAD; Impala requires ADD PARTITION first.
, SHOW PARTITIONS tablename - Answers - Command used to view all existing
partitions on a table.
When NOT to partition - Answers - Avoid partitioning on high-cardinality columns (e.g.,
first name) to prevent creating too many tiny files.
When TO partition - Answers - Partition when full scans are slow, queries frequently
filter on that column, and distinct values are reasonable.
Tools with built-in partition support - Answers - Hive and Impala natively manage
partitions; Sqoop supports partitioned writes via CLI flags.
Apache Flume - Answers - A high-performance system for collecting, aggregating, and
streaming large log and event data into HDFS.
Flume's origin - Answers - Created by Cloudera, donated to Apache in 2011, and
became a top-level project in 2012.
Flume NG vs Flume OG - Answers - NG refers to Next Generation (current
architecture); OG refers to the legacy Original Generation.
Flume's three design goals - Answers - Reliability, Scalability, and Extensibility.
How Flume achieves reliability - Answers - Uses transactional transfers through
Channels, rolling back and retrying on failures.
How Flume achieves scalability - Answers - Scales horizontally by adding more
machines and agents to handle increased load.
How Flume achieves extensibility - Answers - Allows developers to create custom
Sources and Sinks for new storage layers or platforms.
Flume Event - Answers - The basic data unit in Flume, consisting of a payload body
and metadata headers.
Source (Flume component) - Answers - Receives events from external producers and
writes them to one or more Channels.
Sink (Flume component) - Answers - Drains events from a Channel and delivers them
to an external destination.
Channel (Flume component) - Answers - Buffers events between the Source and the
Sink.
Agent (Flume component) - Answers - A JVM process hosting and managing Sources,
Channels, and Sinks.
Partitioning (data storage) - Answers - Subdividing table storage into HDFS
subdirectories by column value to avoid full scans.
Full table scan - Answers - Default non-partitioned behavior where every file in the
directory is read during analysis.
Partition column - Answers - A virtual column whose value is encoded in the directory
name, not in the data file.
Partition directory naming format - Answers - column=value format, e.g., state=CA or
call_date=2014-10-02.
Dynamic partitioning - Answers - Automatic creation of partitions at load time based on
the last column in SELECT/INSERT.
Static partitioning - Answers - Manually specifying the target partition using ALTER
TABLE ... ADD PARTITION.
hive.exec.dynamic.partition - Answers - Hive property that must be set to true to
enable dynamic partitioning.
hive.exec.dynamic.partition.mode - Answers - Hive property set to nonstrict to allow all
partitions in a query to be dynamic.
hive.exec.max.dynamic.partitions.pernode - Answers - Hive limit on dynamic partitions
created per node in a query; default is 100.
hive.exec.max.dynamic.partitions - Answers - Hive limit on total dynamic partitions
created per HiveQL statement; default is 1000.
hive.exec.max.created.files - Answers - Hive limit on total files created across all
nodes by a query; default is 100000.
MSCK REPAIR TABLE - Answers - Hive command that recovers partition metadata
created outside Hive (e.g., via Spark or MapReduce).
LOAD DATA INPATH ... PARTITION(...) - Answers - Hive command that moves an
HDFS file into the specified partition subdirectory.
Hive-only LOAD shortcut - Answers - Hive automatically creates non-existent partition
metadata and subdirectories on LOAD; Impala requires ADD PARTITION first.
, SHOW PARTITIONS tablename - Answers - Command used to view all existing
partitions on a table.
When NOT to partition - Answers - Avoid partitioning on high-cardinality columns (e.g.,
first name) to prevent creating too many tiny files.
When TO partition - Answers - Partition when full scans are slow, queries frequently
filter on that column, and distinct values are reasonable.
Tools with built-in partition support - Answers - Hive and Impala natively manage
partitions; Sqoop supports partitioned writes via CLI flags.
Apache Flume - Answers - A high-performance system for collecting, aggregating, and
streaming large log and event data into HDFS.
Flume's origin - Answers - Created by Cloudera, donated to Apache in 2011, and
became a top-level project in 2012.
Flume NG vs Flume OG - Answers - NG refers to Next Generation (current
architecture); OG refers to the legacy Original Generation.
Flume's three design goals - Answers - Reliability, Scalability, and Extensibility.
How Flume achieves reliability - Answers - Uses transactional transfers through
Channels, rolling back and retrying on failures.
How Flume achieves scalability - Answers - Scales horizontally by adding more
machines and agents to handle increased load.
How Flume achieves extensibility - Answers - Allows developers to create custom
Sources and Sinks for new storage layers or platforms.
Flume Event - Answers - The basic data unit in Flume, consisting of a payload body
and metadata headers.
Source (Flume component) - Answers - Receives events from external producers and
writes them to one or more Channels.
Sink (Flume component) - Answers - Drains events from a Channel and delivers them
to an external destination.
Channel (Flume component) - Answers - Buffers events between the Source and the
Sink.
Agent (Flume component) - Answers - A JVM process hosting and managing Sources,
Channels, and Sinks.