Support Questions
Find answers, ask questions, and share your expertise

/usr/lib/hadoop-mapreduce/hadoop-streamingxxxx.jar not found

Contributor

Hi all,

I was following this tutorial:

http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/

and I couldn't find hadoop-streamingxxxx.jar. I'm using a cluster with hdp-2.3.4.0-3485.

Does any know where to find it or how to add it ?

Thanks ๐Ÿ™‚

1 ACCEPTED SOLUTION

Accepted Solutions

Mentor

@Zaher Mahdhi

you can find streaming jar in /usr/hdp/current/hadoop-mapreduce-client, make sure mapreduce, hdfs and yarn clients are installed on your machine.

[root@sandbox hadoop-mapreduce-client]# ls hadoop-streaming*
hadoop-streaming-2.7.1.2.3.2.0-2950.jar  hadoop-streaming.jar
[root@sandbox hadoop-mapreduce-client]# pwd
/usr/hdp/current/hadoop-mapreduce-client

View solution in original post

8 REPLIES 8

Mentor

@Zaher Mahdhi

you can find streaming jar in /usr/hdp/current/hadoop-mapreduce-client, make sure mapreduce, hdfs and yarn clients are installed on your machine.

[root@sandbox hadoop-mapreduce-client]# ls hadoop-streaming*
hadoop-streaming-2.7.1.2.3.2.0-2950.jar  hadoop-streaming.jar
[root@sandbox hadoop-mapreduce-client]# pwd
/usr/hdp/current/hadoop-mapreduce-client

View solution in original post

Mentor

You can install it using Ambari, go to the host page and add mapreduce client, doesn't hurt to add yarn and hdfs client also

Contributor

@Artem Ervits thanks ๐Ÿ™‚

@Zaher Mahdhi

I see that you are using HDP 2.3.4.0-3485

See the following

[root@phdns01 ~]# find / -name hadoop-streaming*

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming.jar

/usr/hdp/2.3.4.0-3485/oozie/share/lib/mapreduce-streaming/hadoop-streaming-2.7.1.2.3.4.0-3485.jar

[root@phdns01 ~]#

Contributor

Contributor

When testing the example I got this error ๐Ÿ˜• :

16/03/01 01:57:29 INFO mapreduce.Job: Task Id : attempt_1456773989186_0006_r_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
    at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:134)
    at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:459)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

@Zaher Mahdhi Open this as new question to avoid confusion

Mentor

@Zaher Mahdhi I agree, can you please post this as a new question and provide steps to reproduce the problem.