Apache Hadoop

Setup Apache Hadoop on Windows, Linux & Mac OS ..

Apache Hadoop

x

x

Windows & MacOS

1. Download the Apache-Hadoop prebuilt image from Docker Hub repository:

Windows

AMD-based chipset (Intel):

docker pull jporeilly/apache-hadoop:amd

MacOS

ARM-based (Mac M series):

docker pull jporeilly/apache-hadoop:arm

These images are about 14GB .. so please be patient ..!

1. Once completed, deploy the Apache-Hadoop containers.

docker run -it -p 9870:9870 -p 8095:8088 -p 9864:9864 --name AHW jporeilly/apache-hadoop:amd

The YARN Resource Manager port is mapped to: port:8095 to prevent conflict with Cedalo Management Center - MQTT

1. Once completed a shell will open:

root@955b8d17f170:/#

1. Enter: init

init

This stops all running processes, formats the HDFS namenodes, and starts all processes.

1. Enter: jps

jps

Now you can access the Hadoop services at:

• NameNode Web UI: http://localhost:9870

• YARN ResourceManager: http://localhost:8095 (instead of 8088)

• DataNode Web UI: http://localhost:9864

This completes the installation of all tools required for the Big Data course.

Just type exit to exit from the container - this will stop the container.

To start the container enter the following:

docker start -ai AHW

Once the Docker shell opens, just type restart to restart all processes.

NameNode

NameNode

The NameNode is the master node and central component of Hadoop's Distributed File System (HDFS). It acts as the "brain" of the file system.

1. Log into NameNode:

http://localhost:9870localhost

1. You can upload files to the root directory:

YARN

DataNode

Linux