STEPS TO RUN WORD COUNT MAP-REDUCE

If you have hadoop installed (extracted hadoop tar file) please ignore the below installation step.

Installation:

==> please replace user value with appropriate user name from your system
terminal> cd /home/user

==> to download the tarball from below location you:
Either you can paste below link in web browser which would start downloading or you can you wget command to download hadoop tar file.

terminal>wget  http://redrockdigimark.com/apachemirror/hadoop/common/stable/hadoop-2.9.1.tar.gz
http://redrockdigimark.com/apachemirror/hadoop/common/stable/hadoop-2.9.1.tar.gz

==> untar the file using below command
terminal>tar -xzf hadoop-2.9.1.tar.gz

==> you can use the below command to check if the files has been sucessfully extracted or not 
terminal> cd hadoop-2.9.1/bin

==> below ls command will list all the commands which related to hadoop
terminal> ls

==> below command would show you the current files from the current directory.
terminal> /home/user/hadoop-2.9.1/hadoop fs -ls

You can install hadoop in sudo-distributed mode as well. But for the beginning above steps are sufficient.

Creating Jar to be run 

Currently I am using Intellij (IDE) to show you to create jar, You can use any IDE.(Intellij and Eclipse are most frequently used by organisations )

  • Create a intellij maven project.
  • Create a package with name "org.myorg.hadoop" in the project.
  • Create a new class with name "WordCount"
  • Copy and paste the code  from the below location to "WordCount.java"
    • https://worldofhadoopbigdata.blogspot.com/2018/07/hadoop-map-reduce-word-count-java-code.html
  • Now to create jar go to 
    • File->Project Structure -> Artifacts
    • Click on plus sign -> Jar -> From modules with dependencies

    • Provide the main Class as org.myorg.hadoop.WordCount
    • Click On OK.
    • Now all the setting is being done. Now we will create jar.
    • Click on Build-> build artifacts
    • Click on Action Build
  • This would create a jar in location 
    • ProjectLocation/out/artifacts/{Project_name}_jar/{Project_name}.jar

Running the Jar

Now that the jar has been created. You can run the below command to start running the map-reduce.
  • /home/user/hadoop-2.9.1/hadoop jar /path_to/{Project_name}.jar input_file_location output_file_location
  • In above command :
    • first parameter is input_file_location which is the input file on which word count needs to be performed
    • second parameter is output_file_location is the location at which final output is stored

To check the output we can use below command:

/home/user/hadoop-2.9.1/hadoop fs -ls output_file_location 

this directory would have final output of word-count.

Comments

Post a Comment

Popular posts from this blog

Hadoop calculate maximum temperature explained

Image
Analyzing the Data with Hadoop Using Map Reduce  To take advantage of the parallel processing that Hadoop provides, we need to express our query as a MapReduce job because MapReduce framwork will manage the parallel processing by it self. MapReduce divides the processing into 2 phases - the map phase  the reduce phase  Each phase has input in the form of key value pair, and both phases produces output as key value pair. The output that has been generated by the map phase is given to a reduce phase as an input. It is the programmers responsibility to specifies two functions: the map function  the reduce function Lets take an example where input to the map phase is from the below link which has the NCDC data- https://raw.githubusercontent.com/lmsamarawickrama/Hadoop-MapReduce/master/NCDC%20weather%20files/1901 Using the above data we need to calculate maximum temperature per year. While writing a mapreudce code, We choose a text input...
Read more

Sqoop In Depth

Image
In earlier post we have seen  need of Sqqop ? what are the Sqoop commands ? And what are the file formats Sqoop supports. Now lets dive in into the functional aspect of Sqoop like How does Sqoop imports/exports data from/to RDBMS.. Sqoop Import Sqoop’s import tool will run a MapReduce job that connects to the MySQL database and reads the table. By default, this will use four map tasks in parallel.Each task will write its imported results to a different file.If a distributed Hadoop cluster is being used, localhost (in jdbc:mysql://localhost/dbname )should not be specified in the connect string, because map tasks not running on the same machine as the database will fail to connect. For example we have 3 nodes (machines) in a cluster with ip 192.168.0.1, 192.168.0.2, 192.168.0.3 (consider these ip addresses just for example purpose). Lets assume MySql is installed on machine 192.168.0.2 .After running sqoop import with connection string as " jdbc:mysql://localhost/dbname...
Read more

Hadoop Map-Reduce Word Count Java Example

This hadoop tutorial aims to give developers a great start in the world of hadoop mapreduce programming by giving them a hands-on experience in developing their first hadoop based WordCount application. Hadoop MapReduce WordCount example is a standard example where hadoop developers begin their hands-on programming with. This tutorial will help hadoop developers learn how to implement WordCount example code in MapReduce to count the number of occurrences of a given word in the input file. (so called) Pre-requisites to follow this Hadoop WordCount Example Tutorial Hadoop must be installed or you should have a sandbox running on your Virtualbox (or VMWare). In case you have installed Hadoop on your machine Single node hadoop cluster must be configured and running. Optional - IDE must be installed (IntelliJ or Eclipse or any IDE) Hadoop Map Reduce Example - Word Count – How it works? Hadoop WordCount operation occurs in 3 stages – Mapper Phase Shuffle Phase Reducer Ph...
Read more