• Wrong document? Swap it for free
  • Written by students who passed
  • Immediately available after payment
  • Read online or as PDF
Sell
Where do you study
Your language
Document preview thumbnail
Preview 3 out of 21 pages
Exam (elaborations)

BigDataEx1

Document preview thumbnail
Preview 3 out of 21 pages

What are the 5 Phases of Real-Time? - answer-1) Data Distillation 2) Model Development 3) Validation and Deployment 4)real-time scoring 5) model refresh SQOOP - answer--SQL+Hadoop = sq oop -To import data from relational databases into Hadoop and -to export data to relational databases from Hadoop. Apache Hive? - answer--data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. -used to manipulate data What is Apache Hive built on? - answer-Built on top of Apache Hadoop for providing data summarization, query, and analysis What does Hive provide? - answer-Provides a SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop. -Traditional SQL queries must be implemented in the MapReduce Java API to execute SQL applications and queries over distributed data. Apache Spark - answer--A super fast, in-memory data processing engine. -Open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkley's AMPLab, and open sourced in 2010 as an Apache project. -Industry standard for real time streaming data analytics -can combine sql, streaming, and complex analytics Advantages of Spark? - answer--Spark has several advantages compared to other big data and MapReduce technologies like Hadoop and Storm: Speed, Java/ scala/ python, and streaming, ML, visualization Spark Components? - answer--Spark SQL: Fast engine for Hive Interactive Queries -Spark Streaming: Real Time data analysis -MLLiB: Machine learning algorithms -GraphX: Graph Processing Algorithms -Has an advanced DAG execution engine (Apache Spark Core Engine) -Comes with built in operators Metastore - answer-stores metadata; stored in traditional RDBMS format Driver - answer-Compiles, optimizes, like a controller that receives Hive statements DAG (In Optimizer) - answer--Directed Acyclic Graph -In Spark DAG, every edge directs from earlier to later in the sequence. On the calling of Action, the created DAG submits to DAG Scheduler which further splits the graph into the stages of the task. Thrift Server - answer-Allows external clients to interact with hive Ser De - answer-Serializer / Deserializer -The deserializer interface takes a string or binary representation of a record, and translates it into a Java object that Hive can Manipulate. -The Hive serializer will take this Java object, convert it into a suitable format that can be stored into HDFS. What is a live data stream? - answer--includes wide variety of data such as log files generated by customers using your mobile or web applications, e-commerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services, and telemetry from connected devices or instrumentation in data. Cyclic graph? - answer-is a directed graph which contains a path form at least one node back to itself. In simple terms cyclic graphs contain a cycle Whats an acyclic graph? - answer-it is a directed graph which contains absolutely no cycle, that is no node can be traversed back to itself. Pi Estimation Method - answer-The pi sample uses a statistical (quasi-Monte Carlo) method to estimate the value of pi. Points are placed at random in a unit square. The square also contains a circle. The probability that the points fall within the circle are equal to the area of the circle, pi/4. the value of pi can be estimated from the value of 4R. R is the ratio of the number of points that are inside the circle to the total number of points that are within the square. The larger the sample of points used, the better the estimate is. What are the four Vs of big data? what is the fifth? - answer--Volume -Variety -Velocity -Veracity Fifth is Value Volume - answer-Scale of data Velocity - answer--Analysis of streaming data -speed of data Variety - answer--Different forms of data -Diversity of data Veracity - answer-uncertainty of data Value - answer-The ability to achieve greater insights from data when was the beginning of the digital age? - answer-2002 The main components and ecosystem of big data as: - answer--Techniques for analyzing data, such as A/B testing, machine learning and natural learning processing -Big Data technologies, like business intelligence, cloud computing and databases -Visualization, such as charts, graphs and other displays of the data. how much will NEX crop invest to set up an analytics center in Noida? - answer-10 million. 3 years set timeframe. How much does India need in infrastructure investment? - answer-$1.5 trillion in just the next 10 years to help modernize its economy and lift more of its people out of poverty. What do technologies like Hadoop do? - answer-give you the scale and flexibility to store data before you know how you are going to process it. What technologies enable you to run queries without changing the data structures underneath? - answer-MapReduce, Hive, and Impala. What is real-time big data about? - answer-Not just a process for storing petabytes or exabytes of data in a data warehouse, it's about the ability to make better decisions and take meaningful actions at the right time. It about detecting fraud while someone is swiping a credit card, or triggering an offer while a shopper is standing at a checkout line, or placing an ad on a website while someone is reading a specific article. Its about combining and analyzing data so you can take the right action, at the right time, and at the right place. What are we witnessing in big data now? - answer-an explosion of new techniques for analyzing large data sets. In addition to new capabilities for handling large amounts of data, we're also seeing a proliferation of new technologies designed to handle complex, non-traditional data -- precisely the kinds of unstructured or semi-structured data by social media, mobile communications, customer servicer records, warranties, census reports, sensors, and web logs. Three Dimensions of big data? - answer--Volume, Variety, and Velocity. Druid - answer-A system for scanning tens of billions of records per second. It can query 6 terabytes of in-memory data in 1.4 seconds. Googles System - answer-Google, definitely wants their system to be as fast as possible and they definitely put real-time contraints on the internals of their system to make sure that it gives up on certain approaches very quickly. But overall, the system itself is not real time. It's pretty fast, almost all the time. This is the squishy definition of real time. Real-Time Online algorithms - answer-are constrained by time and space limitations. If you "unbound" them to allow more data, they can no longer function as real-time algorithms. What type of intuition do people need to develop? - answer-which kinds of processing are bounded in time, and which kinds aren't The Apache Drill Project - answer-will address the "squishy" factor by scanning through smaller sets of data very quickly. Drill is the open source cousin of Dremel, a Google tool that rips through larger data sets at blazing speeds and spits out summary results, sidestepping the scale issue. -Drill may be complementary to existing frameworks such as Hadoop. What does Drill do? - answer--Brings big data analytics a step closer to real-time interactive processing, which is def. a step in the right direction -Drill is trying to be more things to more people -- probably at the cost of some performance, that's just mostly due to the different environment. How does data appear? - answer-Data appears in all kinds of ways and people have all kinds of preferences for how they want to express what they want, and what kinds of languages they want to write their queries in. What do both Drill and Dremel do? - answer-Scan data in parallel What is a big issue in many situations involving big data? - answer-random failures and resulting data loss What happens when you have lots of data moving across multiple networks and many machines? - answer-theres a greater chance that something will break and portions of the data won't be available. What did Cloudera create? - answer-A system of tools, including Flume and SQOOP, which handle ingestion from multiple sources from multiple sources into Hadoop, and Impala, which enables real-time, ad hoc querying of data. What was the process before Impala? - answer-Before Impala, you did machine learning and larger-scale processes in Hadoop, and the ad hoc analysis in Hive, which involves relatively slow batch processing. Alternatively, you can perform the ad hoc analysis against a traditional database system, which limits your ad-hoc exploration to the data that is captured and loaded into the pre-defined schema. So essentially you are doing machine learning on one side, ad hoc querying on the other side, and then correlating the data between the two systems. What does Impala do? - answer-Enables ad hoc SQL analysis directly on top of your big data systems. You don't have to define the schema before you load the data. Ad Hoc queries in real-time vs traditional analytics - answer-It will take you longer to recognize and respond to new kinds of fraud if you use traditional analytics on top of a traditional enterprise data warehouse, than it would if you had the capabilities to run ad hoc queries in real time. When dealing with fraud, every lost minute translates into lost money.

Content preview

BIGDATA EXAM1 AND ANSWERS 2024

GFS - answer--Google File System
-designed to solve the issues with distributed systems

What does GFS store? - answer-stores large volume of data, and distributed MapReduce
processes that data

When was Hadoop published? - answer-2003-2004. Based on the solution used by
Google in the 1990s

What led to Hadoop - answer-Doug Cutting's Open source project: Nutch

What is the Hadoop solution? - answer--bring computation to the data rather than
bringing data to the computation
-Distribute computing to where data is stored
-run computations where data resides

Apache's Hadoop Definition - answer-The Apache Hadoop software library is a framework
that allows for the distributed processing of large scale data sets across clusters of
computers using simple programming models. It is designed to scale up from single
servers to thousands of machines, each offering local computation and storage. Rather
than rely on hardware to deliver high availability, the library itself is designed to detect
and handle failures at the application layer, so delivering a highly available service on
top of a cluster of computers, each of which may be prone to failures.

What does Hadoop Include? - answer--Hadoop Distributed File System (HDFS)
-Hadoop YARN
-Hadoop Common

Hadoop Distributed File System (HDFS)? - answer--A distributed file system that provides
high-throughput access to application data
-is a distributed, scalable, and portable file system written in Java for the Hadoop
framework
-can store any type of file
-data is automatically split into chunks and replicated for high availability

Hadoop YARN? - answer-- A framework for job scheduling and cluster resource
management
-Hadoop MapReduce: a YARN-based system for parallel processing of large data sets
-manages the cluster resources, for job processing

Hadoop Common? - answer-The common utilities that support the other Hadoop modules

Hadoop- Processing, storing, and analyzing large volumes of data - answer--Software:
handles distribution of data, handling failures
-Hardware: handles storage of data and processing power

,Hadoop is distributed - answer-a Hadoop cluster can have several machines

Hadoop is scalable - answer-can add more machines to the cluster (proportionally adds
capacity)

Hadoop is Fault-Tolerant - answer-can recover hardware failures
-Master re-assigns work
-Data replicates by default on 3 machines
-Nodes that recover rejoin the cluster automatically

Hadoop is Open Source - answer--overseen by Apache
-close to 100 committers from companies like Cloudera, Hortonworks, etc.

Hadoop tools? - answer--ETL(extract, transform, load)
-BI
-Data Storage
-Predictive and Statistical Modeling
-Machine Learning
-others

Hadoop MapReduce - answer--processing framework to process the data
-other processing frameworks, also now available.
- A MapReduce job usually splits the input data-set into independent chunks which are
processed by the map tasks in a completely parallel manner. The framework sorts the
outputs of the maps, which are then input to the reduce tasks. Typically both the input
and the output of the job are stored in a file-system.

MapReduce Process - answer-usually splits the input data-set into independent chunks
which are processed by the map tasks in a completely parallel manner. The framework
sorts the outputs of the map, which are then the input to the reduce task. Typically both
the input and the output of the job are stored in a file-system.

Japan is seeking ________, while India craves _____________ and ___________. The leaders of
both countries, _______________(india) and _______________(Japan), are also working to
counter the growing regional influence of _________ -- an important economic partner to
both but also historically a rival. - answer-Japan is seeking growth markets, while India
craves Advanced technology and Foreign Investment. The Shinzo Abe (Japan), are also
working to counter the growing regional influence of Chine -- an important economic
partner to both but also historically a rival.

Reports suggest that big data and analytics market in India will grow approximately __
times, to ______ by 2020 - answer-8 Times
$16 Billion

Japanese companies are using India as a _____________ ______ to expand into Africa, and
service providers are expanding from Japan into India - answer-Manufacturing base

Hadoop "Ecosystem" - answer--Tools built around the core Hadoop
-All ecosystem tools are open source

, -Tools are designed to extend Hadoop's Functionality
-New tools are added all the time

Hadoop Ecosystem projects included in Cloudera's CDH: - answer--Spark, Hbase, Hive,
Impala, Parquet, Sqoop, Flume/ Kafka, Solr, Hue, Sentry

Spark - answer-in-memory and Streaming processing framework

HBase - answer-noSQL database built on HDFS

Hive - answer-SQL processing engine designed for batch workloads

Impala - answer-SQL Query Engine designed for BI workloads

Parquet - answer-Columnar data storage format

Sqoop - answer-Data movement/ETL to and from RDBMS

Flume, Kafka - answer-streaming data ingestion

Solr - answer-test search functionality

Hue - answer-web based user interface for Hadoop

Sentry - answer-an authorization tool for managing security

Hadoop Is? - answer--Scalable, for parallel/distributable problems (no dependencies
across data)
-A write once, read many solution (vs. RDMS for write and update a lot)

Hadoop is not? - answer--Database (random Access)
-Interactive OLAP (for the moment)
-Updates to files
-Nonparallel work
-Many small files
-Low latency

What do most organizations prefer? - answer--An enterprise-ready distribution of Hadoop
that is: Tested thoroughly, supported, and integrates well with Hadoop projects and
other key software like ETL tools and databases.

Most widely used enterprise-ready Hadoop distributions? - answer-Cloudera,
Hortonworks, and MapR

A cluster? - answer-a group of computers working together

a node? - answer-is an individual computer in that cluster

Two kind of nodes? - answer--Master node (Name Node)

Document information

Uploaded on
September 6, 2024
Number of pages
21
Written in
2024/2025
Type
Exam (elaborations)
Contains
Questions & answers
$13.49

Wrong document? Swap it for free Within 14 days of purchase and before downloading, you can choose a different document. You can simply spend the amount again.
Written by students who passed
Immediately available after payment
Read online or as PDF

Seller avatar
Reputation scores are based on the amount of documents a seller has sold for a fee and the reviews they have received for those documents. There are three levels: Bronze, Silver and Gold. The better the reputation, the more your can rely on the quality of the sellers work.
TOPDOCTOR
5.0
(1)
Sold
10
Followers
5
Items
3387
Last sold
1 year ago



Why students choose Stuvia

Created by fellow students, verified by reviews

Quality you can trust: written by students who passed their tests and reviewed by others who've used these notes.

Didn't get what you expected? Choose another document

No worries! You can instantly pick a different document that better fits what you're looking for.

Pay as you like, start learning right away

No subscription, no commitments. Pay the way you're used to via credit card and download your PDF document instantly.

Student with book image

“Bought, downloaded, and aced it. It really can be that simple.”

Alisha Student

Working on your references?

Create accurate citations in APA, MLA and Harvard with our free citation generator.

Working on your references?

Frequently asked questions