STEPS TO RUN WORD COUNT MAP-REDUCE

If you have hadoop installed (extracted hadoop tar file) please ignore the below installation step.

Installation:

==> please replace user value with appropriate user name from your system
terminal> cd /home/user

==> to download the tarball from below location you:
Either you can paste below link in web browser which would start downloading or you can you wget command to download hadoop tar file.

terminal>wget  http://redrockdigimark.com/apachemirror/hadoop/common/stable/hadoop-2.9.1.tar.gz
http://redrockdigimark.com/apachemirror/hadoop/common/stable/hadoop-2.9.1.tar.gz

==> untar the file using below command
terminal>tar -xzf hadoop-2.9.1.tar.gz

==> you can use the below command to check if the files has been sucessfully extracted or not 
terminal> cd hadoop-2.9.1/bin

==> below ls command will list all the commands which related to hadoop
terminal> ls

==> below command would show you the current files from the current directory.
terminal> /home/user/hadoop-2.9.1/hadoop fs -ls

You can install hadoop in sudo-distributed mode as well. But for the beginning above steps are sufficient.

Creating Jar to be run 

Currently I am using Intellij (IDE) to show you to create jar, You can use any IDE.(Intellij and Eclipse are most frequently used by organisations )

  • Create a intellij maven project.
  • Create a package with name "org.myorg.hadoop" in the project.
  • Create a new class with name "WordCount"
  • Copy and paste the code  from the below location to "WordCount.java"
    • https://worldofhadoopbigdata.blogspot.com/2018/07/hadoop-map-reduce-word-count-java-code.html
  • Now to create jar go to 
    • File->Project Structure -> Artifacts
    • Click on plus sign -> Jar -> From modules with dependencies

    • Provide the main Class as org.myorg.hadoop.WordCount
    • Click On OK.
    • Now all the setting is being done. Now we will create jar.
    • Click on Build-> build artifacts
    • Click on Action Build
  • This would create a jar in location 
    • ProjectLocation/out/artifacts/{Project_name}_jar/{Project_name}.jar

Running the Jar

Now that the jar has been created. You can run the below command to start running the map-reduce.
  • /home/user/hadoop-2.9.1/hadoop jar /path_to/{Project_name}.jar input_file_location output_file_location
  • In above command :
    • first parameter is input_file_location which is the input file on which word count needs to be performed
    • second parameter is output_file_location is the location at which final output is stored

To check the output we can use below command:

/home/user/hadoop-2.9.1/hadoop fs -ls output_file_location 

this directory would have final output of word-count.

Comments

Post a Comment

Popular posts from this blog

Hadoop calculate maximum temperature explained

Image
Analyzing the Data with Hadoop Using Map Reduce  To take advantage of the parallel processing that Hadoop provides, we need to express our query as a MapReduce job because MapReduce framwork will manage the parallel processing by it self. MapReduce divides the processing into 2 phases - the map phase  the reduce phase  Each phase has input in the form of key value pair, and both phases produces output as key value pair. The output that has been generated by the map phase is given to a reduce phase as an input. It is the programmers responsibility to specifies two functions: the map function  the reduce function Lets take an example where input to the map phase is from the below link which has the NCDC data- https://raw.githubusercontent.com/lmsamarawickrama/Hadoop-MapReduce/master/NCDC%20weather%20files/1901 Using the above data we need to calculate maximum temperature per year. While writing a mapreudce code, We choose a text input format (which is a
Read more

Sqoop In Depth

Image
In earlier post we have seen  need of Sqqop ? what are the Sqoop commands ? And what are the file formats Sqoop supports. Now lets dive in into the functional aspect of Sqoop like How does Sqoop imports/exports data from/to RDBMS.. Sqoop Import Sqoop’s import tool will run a MapReduce job that connects to the MySQL database and reads the table. By default, this will use four map tasks in parallel.Each task will write its imported results to a different file.If a distributed Hadoop cluster is being used, localhost (in jdbc:mysql://localhost/dbname )should not be specified in the connect string, because map tasks not running on the same machine as the database will fail to connect. For example we have 3 nodes (machines) in a cluster with ip 192.168.0.1, 192.168.0.2, 192.168.0.3 (consider these ip addresses just for example purpose). Lets assume MySql is installed on machine 192.168.0.2 .After running sqoop import with connection string as " jdbc:mysql://localhost/dbname
Read more

Introduction to Sqoop Part 1

Introduction To use Hadoop for analytics, It is required to load the data into Hadoop clusters. Then later we can use this data for processing it  using traditional processing tool (e.g Map-Reduce/Hive/Pig). Sqoop is  used to Import data from RDBMS system to Hadoop distributed File system (HDFS). And for Exporting data from HDFS back to RDBMS, Sqoop is used. Loading GBs and TBs of data into HDFS from production databases or accessing it from map-reduce applications is a challenging task. While doing so, we have to consider things like data consistency, overhead of running these jobs on production systems and at the end if this process would be efficient or not. Using batch scripts to load data is an inefficient way to go with. Sqoop (“SQL-to-Hadoop”) is a straightforward command-line tool with the following capabilities: Imports individual tables or entire databases to files in HDFS Generates Java classes to allow you to interact with your imported data Provides the a
Read more