Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Apache Hadoop

Setup Apache Hadoop on Windows, Linux & Mac OS ..

The following steps are intended for setting up a Pentaho Lab environment and need to be completed in order to complete the Workshops.

Ensure you have downloaded the Workshop--Installation:

cd
git clone https://github.com/jporeilly/Workshop--Installation

To install git:

sudo apt install git

Prerequisites

  • Ubuntu 24.04 LTS system (physical or virtual machine)

  • User account with sudo privileges

  • Internet connection

  • Basic familiarity with Linux command line

x

In pseudo-distributed mode, all Hadoop daemons (NameNode, DataNode, ResourceManager, NodeManager) run on a single machine. This setup is configured to mimic a multi-node cluster, allowing you to test HDFS operations and run MapReduce or YARN applications as if you had a small cluster. It's the ideal starting point for learning Hadoop.

  1. Update system packages.

sudo apt update && sudo apt upgrade -y

  1. Hadoop is built on Java so you need a Java Development Kit (JDK) installed. Confirm installation.

java --version

  1. Before you begin ensure Docker & Docker Compose have been installed & configured.

docker-compose --version

Create Directories

  1. Create directory structure.

// Some code

  1. Run the docker containers using docker-compose

cd
cd ~/Hadoop
docker-compose up -d
[+] Running 28/5
 ✔ datanode Pulled                                                        32.7s 
 ✔ namenode Pulled                                                        32.6s 
 ✔ nodemanager1 Pulled                                                    32.5s 
 ✔ resourcemanager Pulled                                                 32.3s 
 ✔ historyserver Pulled                                                   32.5s 
[+] Running 9/9
 ✔ Network hadoop_default                Creat...                          0.5s 
 ✔ Volume "hadoop_hadoop_datanode"       Created                           0.0s 
 ✔ Volume "hadoop_hadoop_historyserver"  Created                           0.0s 
 ✔ Volume "hadoop_hadoop_namenode"       Created                           0.0s 
 ✔ Container datanode                    Started                           3.8s 
 ✔ Container namenode                    Started                           3.9s 
 ✔ Container nodemanager                 Starte...                         3.9s 
 ✔ Container historyserver               Star...                           3.8s 
 ✔ Container resourcemanager             St...                             3.9s 
...

Access the Cluster

  1. Can login into any node by specifying the container.

  1. Navigate to mapped data volume.

Accessing the UI

The Namenode UI can be accessed at:

http://localhost:9870/dfshealth.html#tab-overviewlocalhost

ResourceManager UI can be accessed at:

http://localhost:8088/localhost

History Server UI can be accessed at:

http://localhost:8188/applicationhistorylocalhost

Shutdown Cluster

To shut down the cluster.

Time to check we can run some Hadoop Jobs.

So we're going to run a Job that counts the number of instances a word appears in the Canterbury Tales.

Test - Word Count Algorithm

  1. List all the files in our HDFS system.

  1. Create a /user/root/ file.

  1. Verify directory.

  1. Download the hadoop-mapreduce-examples-3.2.1-sources.jar file

We will use a .jar file containing the classes needed to execute MapReduce algorithm.

LogoCentral Repository: org/apache/hadoop/hadoop-mapreduce-examples/3.2.1repo1.maven.org

  1. Save - hadoop-mapreduce-examples-3.2.1-sources.jar to: ~/Hadoop

  2. Download & Save text file - Canterbury Tales or Ulysses

  3. Copy the files into the namenode container.

  1. Create the Input folder.

  1. Copy over /tmp/pg2383.txt to /user/root/input.

10 . Run MapReduce

  1. View the output.

  1. Check the results accessing to the output folder.

  1. Output the text file.

  1. Download the Apache-Hadoop prebuilt image from Docker Hub repository:

Windows

AMD-based chipset (Intel):

MacOS

ARM-based (Mac M series):

These images are about 14GB .. so please be patient ..!

  1. Once completed, deploy the Apache-Hadoop containers.

The YARN Resource Manager port is mapped to: port:8095 to prevent conflict with Cedalo Management Center - MQTT

  1. Once completed a shell will open:

  1. Enter: init

This stops all running processes, formats the HDFS namenodes, and starts all processes.

  1. Enter: jps

Now you can access the Hadoop services at:

This completes the installation of all tools required for the Big Data course.

Just type exit to exit from the container - this will stop the container.

To start the container enter the following:

Once the Docker shell opens, just type restart to restart all processes.

NameNode

The NameNode is the master node and central component of Hadoop's Distributed File System (HDFS). It acts as the "brain" of the file system.

  1. Log into NameNode:

http://localhost:9870localhost

  1. You can upload files to the root directory:

YARN

YARN acts as the operating system for Hadoop clusters by separating resource management from job scheduling and monitoring, allowing multiple data processing engines like MapReduce, Spark, Hive, and others to run simultaneously on the same cluster.

  1. Log into YARN:

http://localhost:8095localhost

DataNode

A DataNode in Hadoop is a worker node in the Hadoop Distributed File System (HDFS) that stores the actual data blocks and serves read/write requests from clients. DataNodes communicate regularly with the NameNode through heartbeat messages to report their health status and the blocks they're storing.

They handle data replication by creating multiple copies of blocks across different nodes to ensure fault tolerance, and they perform block verification to detect corruption. DataNodes also participate in data pipeline operations during file writes and coordinate with other DataNodes to maintain data integrity and availability across the distributed cluster.

  1. Log into the DataNode:

http://localhost:9864localhost

  1. Useful for troubleshooting the Node.

PreviousBig Data
NextJupyter Notebook

Last updated

Was this helpful?