Nosql And Big Data 100% Verified
T/F Spark is a database - ANSWER F, it is a query engine.
What does RDD stand for? - ANSWER Resilient Distributed Dataset
T/F A transformation changes a RDD. - ANSWER F, it defines a NEW RDD based on the
current one. RDDs are immutable.
T/F the line mydata.upper() will trigger an execution for an RDD. - ANSWER F, RDDs are
not processed until an action is performed. upper() is a transformation.
T/F RDD Resilience in RDD means that we loose data in memory, we can redo the
transformations based on the RDD lineage. - ANSWER T, you can view the lineage of the
RDD using toDebugString
What is pipelining in Spark - ANSWER When possible, Spark will do row by row
processing of sequences of transformations, so no data is stored.
T/F If have the line "the cow eats grass" as input in our map function, then the
transformation .map(lambda x: x.upper()), will create a new RDD that transform the line
to upper letters. The line in the new RDD would read "THE COW EATS GRASS". -
ANSWER T
T/F Each RDD stores data in memory. - ANSWER F, RDDs do NOT store data.
T/F Spark can work with all types of input file formats. - ANSWER T
T/F RDDs are partitions - ANSWER T, An RDD dataset is a collection of partitioned data.
, Tasks are performed in parallel in each partition.
T/F Spark can only run on a cluster with YARN as Resource Manager software. -
ANSWER F, Spark can run either standalone or with a cluster manager like Yarn but can
also be other managers like Mesos.
T/F In Spark with RDDs, a groupBy is a wide transformation - ANSWER T, data may
reside in multiple partitions. This would require a re-partitioning
What is a narrow transformation in the context of Spark - ANSWER The records required
to compute the record resided in a single partition in the parent RDD (e.g., map, flatMap,
filter)
What is a wide transformation in the context of Spark - ANSWER Data required to
compute records in a partition may reside in multiple partitions of the parent RDD (e.g.,
groupBy, reduceByKey, distinct, join)
T/F If Sparks runs on HDFS, then to each HDFS partition a RDD partition is created. -
ANSWER T, to each HDFS partition a RDD partition is created.
Where does spark process the data - ANSWER Main memory in executor
T/F An execution plans consists of stages. Each stages has a collection of tasks. Each
stage only includes transformations that are narrow. As soon as a wide transformation
is applied, a new stage starts. - ANSWER T, Operations that can run on the same
partition are executed in stages. Tasks within a stage are pipelined together. Every time
re-partition is needed, a new stage starts.
T/F In Spark 3, a reshuffle from a wide transformation will always yield 200 partitions in
the new DataFrame. - ANSWER F, there is a setting in Spark 3 on for adaptive
optimization. `spark.sql.adaptive.enable` to True
T/F Hive is a database. - ANSWER F, Hive is a data warehousing system on top of
T/F Spark is a database - ANSWER F, it is a query engine.
What does RDD stand for? - ANSWER Resilient Distributed Dataset
T/F A transformation changes a RDD. - ANSWER F, it defines a NEW RDD based on the
current one. RDDs are immutable.
T/F the line mydata.upper() will trigger an execution for an RDD. - ANSWER F, RDDs are
not processed until an action is performed. upper() is a transformation.
T/F RDD Resilience in RDD means that we loose data in memory, we can redo the
transformations based on the RDD lineage. - ANSWER T, you can view the lineage of the
RDD using toDebugString
What is pipelining in Spark - ANSWER When possible, Spark will do row by row
processing of sequences of transformations, so no data is stored.
T/F If have the line "the cow eats grass" as input in our map function, then the
transformation .map(lambda x: x.upper()), will create a new RDD that transform the line
to upper letters. The line in the new RDD would read "THE COW EATS GRASS". -
ANSWER T
T/F Each RDD stores data in memory. - ANSWER F, RDDs do NOT store data.
T/F Spark can work with all types of input file formats. - ANSWER T
T/F RDDs are partitions - ANSWER T, An RDD dataset is a collection of partitioned data.
, Tasks are performed in parallel in each partition.
T/F Spark can only run on a cluster with YARN as Resource Manager software. -
ANSWER F, Spark can run either standalone or with a cluster manager like Yarn but can
also be other managers like Mesos.
T/F In Spark with RDDs, a groupBy is a wide transformation - ANSWER T, data may
reside in multiple partitions. This would require a re-partitioning
What is a narrow transformation in the context of Spark - ANSWER The records required
to compute the record resided in a single partition in the parent RDD (e.g., map, flatMap,
filter)
What is a wide transformation in the context of Spark - ANSWER Data required to
compute records in a partition may reside in multiple partitions of the parent RDD (e.g.,
groupBy, reduceByKey, distinct, join)
T/F If Sparks runs on HDFS, then to each HDFS partition a RDD partition is created. -
ANSWER T, to each HDFS partition a RDD partition is created.
Where does spark process the data - ANSWER Main memory in executor
T/F An execution plans consists of stages. Each stages has a collection of tasks. Each
stage only includes transformations that are narrow. As soon as a wide transformation
is applied, a new stage starts. - ANSWER T, Operations that can run on the same
partition are executed in stages. Tasks within a stage are pipelined together. Every time
re-partition is needed, a new stage starts.
T/F In Spark 3, a reshuffle from a wide transformation will always yield 200 partitions in
the new DataFrame. - ANSWER F, there is a setting in Spark 3 on for adaptive
optimization. `spark.sql.adaptive.enable` to True
T/F Hive is a database. - ANSWER F, Hive is a data warehousing system on top of
