Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

/usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

avatar
Rising Star

Hi all,

I was following this tutorial:

http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/

and I couldn't find hadoop-streamingxxxx.jar. I'm using a cluster with hdp-2.3.4.0-3485.

Does any know where to find it or how to add it ?

Thanks 🙂

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Zaher Mahdhi

you can find streaming jar in /usr/hdp/current/hadoop-mapreduce-client, make sure mapreduce, hdfs and yarn clients are installed on your machine.

[root@sandbox hadoop-mapreduce-client]# ls hadoop-streaming*
hadoop-streaming-2.7.1.2.3.2.0-2950.jar  hadoop-streaming.jar
[root@sandbox hadoop-mapreduce-client]# pwd
/usr/hdp/current/hadoop-mapreduce-client

View solution in original post

8 REPLIES 8

avatar
Master Mentor

@Zaher Mahdhi

you can find streaming jar in /usr/hdp/current/hadoop-mapreduce-client, make sure mapreduce, hdfs and yarn clients are installed on your machine.

[root@sandbox hadoop-mapreduce-client]# ls hadoop-streaming*
hadoop-streaming-2.7.1.2.3.2.0-2950.jar  hadoop-streaming.jar
[root@sandbox hadoop-mapreduce-client]# pwd
/usr/hdp/current/hadoop-mapreduce-client

avatar
Master Mentor

You can install it using Ambari, go to the host page and add mapreduce client, doesn't hurt to add yarn and hdfs client also

avatar
Rising Star

@Artem Ervits thanks 🙂

avatar
Master Mentor
@Zaher Mahdhi

I see that you are using HDP 2.3.4.0-3485

See the following

[root@phdns01 ~]# find / -name hadoop-streaming*

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming.jar

/usr/hdp/2.3.4.0-3485/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

[root@phdns01 ~]#

avatar
Rising Star

avatar
Rising Star

When testing the example I got this error 😕 :

16/03/01 01:57:29 INFO mapreduce.Job: Task Id : attempt_1456773989186_0006_r_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
    at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

avatar
Master Mentor

@Zaher Mahdhi Open this as new question to avoid confusion

avatar
Master Mentor

@Zaher Mahdhi I agree, can you please post this as a new question and provide steps to reproduce the problem.