Hadoop Admin Training Institute in Pune

What is Hadoop Admin?

Hadoop is an associate degree open supply package framework designed for storage and process of huge scale type of information on clusters of artifact hardware. The Apache Hadoop software library is a framework that allows the data distributed processing across clusters for computing using simple programming models called Map Reduce. It is designed to rescale from single servers to a cluster of machines and every giving native computation and storage in economical means. It works in a series of map-reduce jobs and each of these jobs is high-latency and depends on each other. So no job can start until the previous job has been finished and successfully completed. Hadoop solutions normally include clusters that are hard to manage and maintain. In many scenarios, it requires integration with other tools like MySQL, mahout, etc. We have another popular framework that works with Apache Hadoop i.e. Spark. Apache Spark allows software developers to develop complex, multi-step data pipeline application patterns. It also supports in-memory data sharing across DAG (Directed Acyclic Graph) based applications, so that different jobs can work with the same shared data. Spark runs on top of the Hadoop Distributed File System (HDFS) of Hadoop to enhance functionality. Spark does not have its own storage so it uses other supported storage. With the capabilities of in-memory data storage and data processing, the spark application performance is more time faster than other big data technologies or applications. Spark has a lazy evaluation which helps with optimization of the steps in data processing and control. It provides a higher-level API for improving productivity and consistency. Spark is designed to be a fast real-time execution engine that works both in memory and on disk. Spark is originally written in Scala language and it runs on the same Java Virtual Machine (JVM) environment. It currently supports Java, Scala, Clojure, R, Python, SQL for writing applications.

Why Should I take Hadoop Admin Training in Pune?

Apache Hadoop framework allows us to write distributed applications or systems. It is more efficient and it automatically distributes the work and data among machines that lead a parallel programming model. Hadoop works with different kinds of data effectively. It also provides a high fault-tolerant system to avoid data losses. Another big advantage of Hadoop is that it is open source and compatible with all platforms as it’s based on java. In the market, Hadoop is the only solution to work on big data efficiently in a distributed manner. The Apache Hadoop software library is a framework that allows the data distributed processing across clusters for computing using simple programming models called Map Reduce. It is designed to proportion from single servers to the cluster of machines and every giving native computation and storage in an economical manner. It works in a series of map-reduce jobs and each of these jobs is high-latency and depends on each other. So no job can start until the previous job has been finished and successfully completed. Hadoop solutions normally include clusters that are hard to manage and maintain. In many scenarios, it requires integration with other tools like MySQL, mahout, etc.

We have another popular framework that works with Apache Hadoop i.e. Spark.

Apache Spark allows software developers to develop complex, multi-step data pipeline application patterns. It also supports in-memory data sharing across DAG (Directed Acyclic Graph) based applications, so that different jobs can work with the same shared data. Spark runs on top of the Hadoop Distributed File System (HDFS) of Hadoop to enhance functionality. Spark does not have its own storage so it uses other supported storage. With the capabilities of in-memory data storage and data processing, the spark application performance is more time faster than other big data technologies or applications. Spark has a lazy evaluation which helps with optimization of the steps in data processing and control. It provides a higher-level API for improving productivity and consistency. Spark is designed to be a fast real-time execution engine that works both in memory and on disk.

Big Data: Is the data with some characteristics:

5V’s of big data: Volume, Variety, Velocity, Veracity, Value

Volume – Size which can be processed through RDBMS like oracle, MySQL.

Variety – Different types of data → Structured, Semi, Unstructured.

Velocity – Data may come at any speed…

Veracity – No Consistency means Uncertain…

Value – Data have some insights…

Hadoop – Its an open-source framework developed using Java and developed to process and store big data ie. large scale data on the cluster of commodity hardware. This framework allows for the distributed processing of large data across the cluster using a simple Programming model called Map Reduce.

Where Hadoop Admin can be used?

Machine Learning – Machine learning is that the scientific study of algorithms and applied mathematics models that pc systems use to perform a particular task while not mistreatment specific directions. AI – Machine intelligence which behaves like a human and takes decisions. Data Mining – Finding meaningful information from raw data using standard methods.

Data Analysis – information analysis could be a method of inspecting, cleansing, transforming and modeling data with the goal of discovering useful information, informing conclusions and supporting decision-making.

Social Network analysis – Facebook, youtube, google, twitter, LinkedIn data analysis. Graph and data visualization – Data representation through graphs, charts, images, etc.

Tools in Hadoop:

•  HDFS (Hadoop Distributed File System) basic storage for Hadoop.
• Apache Pig is an ETL (Extract Transform and Load) tool.
• Map scale back could be a programmatic model engine to execute man jobs.
• Apache Hive is a Data Warehouse tool used to work on Historical data using HQL.
• Apache Sqoop is a tool for Import and export data from RDBMS to HDFS and Vice-Versa.
• Apache Ooozie is a tool for Job scheduling to control applications over the cluster.
• Apache HBase is a NoSQL database based on CAP(Consistency Automaticity Partition) theory.
• A spark could be a framework will in memory computation and works with Hadoop. This framework is based on scala and java language.

Physical architecture: Master-Slave architecture

Cluster – Group of computers connected with each other.

5 Daemons in Hadoop – Services: Hadoop 1.x series framework

Name node, Secondary Name node, Data node, Job tracker, task tracker

Daemons – Background running processes… Thread running at the background

Divide architecture into the following:

Storage/HDFS architecture – Name node, Secondary name node, and Data node, Blocks

Master ->

Name node – Master node which handles and manages all metadata info and status control..

Name node as manager …

Secondary name node – Asst. Manager. Helping node to NN which maintains metadata in FS image file system and log generation etc.. It’s not a backup of NN.

Slave ->

Data Node – It’s a slave node. Where data resides. Data will be stored using the block system.

Blocks are memory blocks. Have some size which is modifiable.

Hadoop 1.x Default Block size is 64MB

Hadoop 2.x Default Block size 128MB

Default replication factor is : 3

1 Tb data ⇒ 3 TB on cluster

5 node cluster where 1 is master and 4 are slaves having 1TB Hdd capacity each.

If you want to store 50GB data with replication 3 then how many blocks will get created on each slave?

150*1024 => 153600MB/4 => 38400MB/each machine => 600 Blocks on each slave

It uses the rack awareness algorithm will take care of blocks allotment.

Process architecture

Job tracker – Master → Receives a client request to perform an operation over the cluster. Emp.txt

Select count(*) from emp;

JobID → 00001100 →

Task Tracker – Slave → Perform a task using MR phases. → heartbeats will be given to JT.

MR2.x the Job Tracker is divided into three services:

Resource Manager- Is a persistent YARN service that receives and runs applications on the cluster. A MapReduce Job is an application.

JobHistoryServer- To provide information about jobs and completions.

Application Master- To manage each MR job and terminate it when completed. Task Tracker is replaced with Node Manager, that manages resources and deployment on a node. It’s also responsible for launching containers that are of MR Task.

Speculative Execution Mechanism -> If a task tracker is failed or not working then JT will assign the same task to other available TT. Rack awareness algorithm – HDFS file distribution over the network in blocks.