Hadoop Certification

For data in motion. Powered by Apache NiFi. 1) real-time - add, trace, adjust; 2) integrated - common input, output, transformation; 3) secure - security rules, encryption, traceability; 4) adaptive - adapts data flow, scalable; if connection poor skinnies down data - answer-Hortonworks Data Flow (HDF) A user-driven process of searching for patterns or specific items in a data set. Data discovery applications use visual tools such as geographical maps, pivot-tables, and heat-maps to make the process of finding patterns or specific items rapid and intuitive. Data discovery may leverage statistical and data mining. Ex. Web log analysis, online ad placement, claims notes mining - answer-Data discovery Ex. sensor data ingest - answer-ETL onboard Ex. individual driver histories - answer-Active archive Perishable insights - answer-Data in motion Historical insights - answer-Data at rest Supports data discovery, single view, predictive analytics - answer-Actionable intelligence A Single View application aggregates data from multiple sources into a central repository to create a single view of anything — of customers, inventory, systems - answer-Single view Offers the leading platform for Operational Intelligence. It enables the curious to look closely at what others ignore—machine data—and find what others never see: insights that can help make your company more productive, profitable, competitive and secure - answer-Splunk An open source big data processing framework built around speed, ease of use, and sophisticated analytics. It was originally developed in 2009 in UC Berkeley's AMPLab, and open sourced in 2010 as an Apache project - answer-Apache Splunk Real-time event processing for sensor and business activity monitoring. A free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for real-time processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language. Ingests millions of events per second. Manage with Ambari. Horizontally scalable. Fixed, low latency and continuous processing for very high frequency streaming data. - answer-Apache Storm Data operating system. Cluster resource management. 2013 - includes batch, interactive and realtime. At core of Hortonworks Data Platform (HDP) for data at rest. Centralized platform for: 1) operations - cluster management, one data lake or clusters; 2) governance - data lifecycle mgt, modeling with metadata, lineage capability 3) security - roles or data tags, encryption at rest and in motion, authentication. Includes data functions for: batch, machine learning, search, interactive, streaming - answer-YARN SQL:2011 for analytics - answer-Hive on YARN Data at rest. Powered by Open Enterprise Hadoop. 1) Open - open source; 2) Central - Yarn at core; 3) Interoperable - existing technology, skills; 4) Ready - enterprise-ready re operations, governance, security; dev efforts include: 1) data management; 2) data access; 3) governance and integration; 4) operations; 5) security - answer-Hortonworks Data Platforms (HDP) An open source cluster computing framework originally developed in the AMPLab at University of California, Berkeley but was later donated to the Apache Software Foundation where it remains today. Integrated component of HDP. Agile analytics using data science notebooks, includes geospatial, entity resolution; wide array of data sources; RDD sharing, HDFS memory tier. Newer approach than SQL handled by Hive. Data access engine for fast, large scale data processing. Designed for iterative, in-memory computations and interactive data mining. APIs for Scala, Java, Python. Spark SQL, Spark Streaming, MLlib, GraphX - can run as a YARN workload - can run on a single data set in Hadoop. - answer-Apache Spark at Scale A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable, partitioned collection of elements that can be operated on in parallel. - answer-Resilient Distributed Dataset (RDD) The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file-system written in Java for the Hadoop framework. A Hadoop cluster has nominally a single namenode plus a cluster of datanodes, although redundancy options are available for the namenode due to its criticality. Each datanode serves up blocks of data over the network using a block protocol specific to HDFS. The file system uses TCP/IP sockets for communication. Clients use remote procedure call (RPC) to communicate between each other. - answer-Hadoop Distributed File System (HDFS) SQL interface to Hadoop data. Most widely used SQL engine in Hadoop Community. Alternative to Spark at Scale. Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. While initially developed by Facebook, Apache Hive is now used and developed by other companies such as Netflix. Amazon maintains a software fork of Apache Hive that is included in Amazon Elastic MapReduce on Amazon Web Services. Enable transactions, SQL:2011 - answer-Apache Hive on YARN 90 that innovate for Hortonworks; customer advocates and roadmapping - answer-Apache Hadoop Committers Includes: integrated customer portal, knowledge base, on-demand training, smartsense (machine learning and predictive analytics on customer cluster); proactively optimizes your cluster - answer-Hortonworks customer support Machine learning is a subfield of computer science that evolved from the study of pattern recognition and computational learning theory in artificial intelligence. In 1959, Arthur Samuel defined machine learning as a "Field of study that gives computers the ability to learn without being explicitly programmed". - answer-Machine learning Includes: answers; knowledge base; code hub; sandbox; tutorials; events - answer-Hortonworks Community Connection - answer-Hortonworks Partnerworks One node, mini-cluster HDP that runs in VM on laptop with tutorials and sample data sets - answer-Hortonworks Sandbox - answer-Hortonworks Blog A useful framework to think about designing big data applications. Nathan Marz designed this generic architecture addressing common requirements for big data based on his experience working on distributed data processing systems at Twitter. Includes Batch Layer, Speed Layer, View Layer. - answer-Lambda architecture Hadoop enables them. - answer-Multi-tenant access and processing Physical, virtual, cloud - answer-Deployment options Five pillars: data management; data access; data governance; security; operations - answer-Hortonworks data platform (HDP) Hive community focus on speed, scale, SQL.; SQL-based interactive query; Included ORC, Tez - answer-Stinger Initiative An intelligent big data file format for Hadoop and Hive. The Optimized Row Columnar (ORC) file format provides a highly efficient way to store Hive data. It was designed to overcome limitations of the other Hive file formats. Using ORC files improves performance when Hive is reading, writing, and processing data. A Framework for YARN-based, Data Processing Applications In Hadoop. Executes queries in memory and eliminates temporal HDFS writes between work units. Apache Tez provides a developer API and framework to write native YARN applications that bridge the spectrum of interactive and batch workloads. It allows those data access applications to work with petabytes of data over thousands nodes. The Apache Tez component library allows developers to create Hadoop applications that integrate natively with Apache Hadoop YARN and perform well within mixed workload clusters. - answer-Apache Tez engine Included beyond read only; ACID allowing insert, update, delete; data encryption; SQL:2011 analytics; cost based optimizer for complex joins; SQL Query editor; LLAP integrated with YARN; transaction support. WIP? - answer-S Sub-second analytical queries in Hive. - answer-Live Long and Process (LLAP) Center of most operations and management of Hadoop. A software project of the Apache Software Foundation aimed at making Hadoop management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs; Includes: Cluster Management (install, deployment, ongoing maintenance), Service Management (don't have to manually start services across all nodes), Configuration Management (don't have to manage a lot of files across every node), Monitoring, Extensibility (Ambari Views, Ambari Stack). Can be used to deploy other systems; adopted by Open Data Platform; - answer-Apache Ambari Can run unsupervised and supervised tasks; - answer-Spark ML (Machine Learning) k-means clustering, or Lloyd's algorithm [2], is an iterative, data-partitioning algorithm that assigns n observations to exactly one of k clusters defined by centroids, where k is chosen before the algorithm starts. The algorithm proceeds as follows: Choose k initial cluster centers (centroid). - answer-K-means Collaborative filtering (CF) is a popular recommendation algorithm that bases its predictions and recommendations on the ratings or behavior of other users in the system. - answer-Collaborative filtering (CF) In machine learning and statistics, dimensionality reduction or dimension reduction is the process of reducing the number of random variables under consideration, via obtaining a set of "uncorrelated" principal variables. It can be divided into feature selection and feature extraction. - answer-Dimensionality reduction Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. - answer-Principal Component Analysis (PCA) In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix. - answer-Singular Value Decomposition (SVD) In machine learning, naive Bayes classifiers are a family of simple probabilistic classifiers based on applying Bayes' theorem with strong (naive) independence assumptions between the features. - answer-Naïve Bayes classification Decision tree learning uses a decision tree as a predictive model which maps observations about an item to conclusions about the item's target value. It is one of the predictive modelling approaches used in statistics, data mining and machine learning. - answer-Decision Tree classification Random forests is a notion of the general technique of random decision forests[1][2] that are an ensemble learning method for classification, regression and other tasks, that operate by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. Random decision forests correct for decision trees' habit of overfitting to their training set - answer-Random Forest classification Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. - answer-Gradient Boosted Tree classification A Support Vector Machine (SVM) is a discriminative classifier formally defined by a separating hyperplane. In other words, given labeled training data (supervised learning), the algorithm outputs an optimal hyperplane which categorizes new examples - answer-SVM linear model The logistic function, also called the sigmoid function was developed by statisticians to describe properties of population growth in ecology, rising quickly and maxing out at the carrying capacity of the environment. It's an S-shaped curve that can take any real-valued number and map it into a value between 0 and 1, but never exactly at those limits. - answer-Logistic regression In statistics, linear regression is an approach for modeling the relationship between a scalar dependent variable y and one or more explanatory variables (or independent variables) denoted X. The case of one explanatory variable is called simple linear regression. For more than one explanatory variable, the process is called multiple linear regression. - answer-Linear regression Distributed database scales horizontally to PB of data. SQL and NoSQL interfaces. For low latency and read-write usage case needs. Facebook messaging, Ebay searching and indexing, Bloomberg time series anal