########-----------INSTALLING HADOOP, HIVE IN WINDOWS & WORKING WITH HDFS -------------- ########
################ ------------ AKULA VENKATA RAMANA ------------ #################
Link to download Java JDK 8: https://www.oracle.com/java/technologies/downloads/#java8 (go for java version 1.8.0_261)
Link to download Hadoop(hadoop - 2.8.0): https://muhammadbilalyar.github.io/blogs/How-to-install-Hadoop-on-Window-10/
HIVE Installtion on WINDOWS:
https://www.youtube.com/watch?v=npyRXkMhrgk
Download Hive -
https://archive.apache.org/dist/hive/hive-2.1.0/
Download Derby Metastore-
https://archive.apache.org/dist/db/derby/db-derby-10.12.1.1/
Download hive-site.xml-
https://drive.google.com/file/d/1qqAo7RQfr5Q6O-GTom6Rji3TdufP81zd/view
# How to extract .tar.gz files on Windows 11
Run cmd as Admin:
tar -xvzf source file path -C destination path
# Format the HDFS filesystem: Run the following command in the admin command prompt to format the HDFS filesystem:
hdfs namenode -format
# Start Hadoop services: Run the following command in the command prompt to start the Hadoop services: start-dfs & start-yarn
start-dfs
start-yarn
# Verify installation: run the following command in the command prompt to verify that the Hadoop cluster is running: jps
C:\hadoop-2.8.0\sbin>jps
16592 ResourceManager
19296 DataNode
15096 Jps
18776 NameNode
27644 NodeManager
if we get all nodes & jps printed on console that means you have now successfully configured Hadoop in pseudo-distributed mode on Windows.
pseudo nodes means: In Hadoop, the pseudo-distributed mode is a single-node Hadoop cluster, where all the Hadoop daemons run on a single machine.
This mode is useful for testing and development purposes when we want to simulate a distributed Hadoop environment on a single machine.
In pseudo-distributed mode, all the Hadoop daemons such as NameNode, DataNode, ResourceManager, and NodeManager run in separate Java Virtual Machines (JVMs)
on the same machine. This means that we can use Hadoop commands and APIs to perform the same tasks as we would in a fully distributed mode cluster.
********************************************** Installation completed *****************************************************************************
What next? Once the Hadoop cluster running.....
# Start Derby > bin
derby-10.12.1.1
# Start the port:
startNetworkServer -h 0.0.0.0
to get the below connections on port 1527
Fri Mar 03 14:51:19 IST 2023 : Security manager installed using the Basic server security policy.
Fri Mar 03 14:51:19 IST 2023 : Apache Derby Network Server - 10.12.1.1 - (1704137) started and ready to accept connections on port 1527
# Start Hive: hive-2.1.0
# If you get this error: C:\hive-2.1.0\bin>hive
"Missing hadoop installation: C:\hadoop-2.8.0\bin must be set" then go to hadoop directory and set the below path:
set HADOOP_HOME=C:\hadoop-2.8.0
Now start Hive:
C:\hive-2.1.0\bin>hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/hive-2.1.0/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/hadoop-2.8.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Connecting to jdbc:hive2://
Connected to: Apache Hive (version 2.1.0)
Driver: Hive JDBC (version 2.1.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.1.0 by Apache Hive
hive>
# Show Databases / Dropping Tables / Dropping Databases etc
show databases;
use bdsassignment;
DROP TABLE ret_table_data;
DROP DATABASE bdsassignment;
# Creating a data base in Hive
create database bdsassignment;
# Use that database
use bdsassignment;
# Creating table in Hive
CREATE TABLE ret_table_data (
record_no INT,
invoice STRING,
stockcode STRING,
description STRING,
quantity INT,
invoicedate TIMESTAMP,
price DOUBLE,
customer_id INT,
country STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
"separatorChar" = ",",
"quoteChar" = "\"",
"escapeChar" = "\\"
)
STORED AS TEXTFILE
TBLPROPERTIES ("skip.header.line.count"="1");
# Now load the data intot the Hive table
load data local inpath 'C:\Assignment -2 2023 BDS DATA SET online_retail_data.csv' into table ret_table_data;
# To check if the data is propely loaded
select * from ret_table_data limit 20;
# For checking the Table description
describe formatted ret_table_data;
# Location where it is stored in hdfs: (check this in hadoop-2.8.0 command)
hdfs://localhost:9001/user/hive/warehouse/bdsassignment.db/ret_table_data
# To check the lehgth of the data set
select count(*) from ret_table_data;
# 2) Country from which the maximum revenue was collected from sales in the month of March 2010
SELECT country, SUM(quantity*price) AS revenue
FROM (
SELECT country, quantity, price,
substr(invoicedate, 1, 10) AS inv_date
FROM ret_table_data
WHERE ((invoicedate LIKE '%03-2010%') OR (invoicedate LIKE '%03/2010%'))
) t
GROUP BY country
ORDER BY revenue DESC
LIMIT 1;
# To monitor the Job status
http://localhost:8088
http://localhost:50070
# add the following line to your mapred-site.xml file, replacing the path with the correct path to your Python executable:
<property>
<name>mapreduce.executable</name>
<value>C:\Users\venka\anaconda3\pkgs\python-3.9.12-h6244533_0\python.exe</value>
</property>
# To leave the safemode
hdfs dfsadmin -safemode leave
# Run the MapReduce job using the following command: ensure to have py files of mapper & reducer on specified path:
# Now run the below command in Hadoop:
hadoop jar C:\hadoop-2.8.0\share\hadoop\tools\lib\hadoop-streaming-2.8.0.jar ^
-files "file:///C:/wordcoutmapper.py,file:///C:/wordcoutreducer.py" ^
-mapper " C:\Users\venka\anaconda3\pkgs\python-3.9.12-h6244533_0\python.exe wordcoutmapper.py" ^
-reducer "C:\Users\venka\anaconda3\pkgs\python-3.9.12-h6244533_0\python.exe wordcoutreducer.py" ^
-input /BDS_Assignment2/word_count_data.txt ^
-output /BDS_Assignment2/result
hadoop jar C:\hadoop-2.8.0\share\hadoop\tools\lib\hadoop-streaming-2.8.0.jar ^
-files "file:///C:/Q1mapper.py,file:///C:/Q1reducer.py" ^
-mapper " C:\Users\venka\anaconda3\pkgs\python-3.9.12-h6244533_0\python.exe Q1mapper.py" ^
-reducer "C:\Users\venka\anaconda3\pkgs\python-3.9.12-h6244533_0\python.exe Q1reducer.py" ^
-input hdfs://localhost:9001/user/hive/warehouse/bdsassignment.db/ret_table_data ^
-output /BDS_Assignment2/result
# Command to list the contents of the /BDS_Assignment2/result directory:
hadoop fs -ls /BDS_Assignment2/result
# Once the MapReduce job completes, you can view the output using the following command:
hadoop fs -cat /BDS_Assignment2/result/part-00000
# To delete the directory /BDS_Assignment2/result in HDFS,
hadoop fs -rm -r /BDS_Assignment2/result
stop-dfs
stop-yarn
Ramana video : https://youtu.be/HmBtvWhs2ng
Ramana linkedin : https://www.linkedin.com/in/akula-v-746ab026
*********************************************************** THANK YOU **************************************************************************
All the messages below are just forwarded messages if some one feels hurt about it please add your comments we will remove the post.Host/author is not responsible for these posts.
