/usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

[#1]

Hi all,

I was following this tutorial:

http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/

and I couldn't find hadoop-streamingxxxx.jar. I'm using a cluster with hdp-2.3.4.0-3485.

Does any know where to find it or how to add it ?

Thanks 🙂

[#2 · Accepted solution]

@Zaher Mahdhi

you can find streaming jar in /usr/hdp/current/hadoop-mapreduce-client, make sure mapreduce, hdfs and yarn clients are installed on your machine.

[root@sandbox hadoop-mapreduce-client]# ls hadoop-streaming*
hadoop-streaming-2.7.1.2.3.2.0-2950.jar  hadoop-streaming.jar
[root@sandbox hadoop-mapreduce-client]# pwd
/usr/hdp/current/hadoop-mapreduce-client

[#3]

You can install it using Ambari, go to the host page and add mapreduce client, doesn't hurt to add yarn and hdfs client also

[#4]

@Artem Ervits thanks 🙂

[#5]

@Zaher Mahdhi

I see that you are using HDP 2.3.4.0-3485

See the following

[root@phdns01 ~]# find / -name hadoop-streaming*

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming.jar

/usr/hdp/2.3.4.0-3485/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

[root@phdns01 ~]#

[#6]

@Neeraj Sabharwal Thank u for your answer

[#7]

When testing the example I got this error 😕 :

16/03/01 01:57:29 INFO mapreduce.Job: Task Id : attempt_1456773989186_0006_r_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
    at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

[#8]

@Zaher Mahdhi Open this as new question to avoid confusion

[#9]

@Zaher Mahdhi I agree, can you please post this as a new question and provide steps to reproduce the problem.