Reply
New Contributor
Posts: 2
Registered: ‎11-20-2016

yarn java configuration issue

I've built a new cluster and all seems right but yarn/mapreduce jobs won't run. Doing a simple wordcount I get...

 

17/09/05 22:44:21 INFO client.RMProxy: Connecting to ResourceManager at cldradn101.dassvm.nccs.nasa.gov/10.71.11.21:8032
17/09/05 22:44:22 INFO input.FileInputFormat: Total input paths to process : 1
17/09/05 22:44:22 INFO mapreduce.JobSubmitter: number of splits:8
17/09/05 22:44:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504648306083_0003
17/09/05 22:44:22 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/yarn/.staging/job_1504648306083_0003
java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.records.ApplicationSubmissionContext.setAMContainerResourceRequests(Ljava/util/List;)V
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:579)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:315)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)

 

 

Can someone suggest where to look for I assume is a java issue? Thanks in advance!

Posts: 1,525
Kudos: 266
Solutions: 232
Registered: ‎07-31-2013

Re: yarn java configuration issue

What CDH version is your installed cluster, and what version are you compiling/bundling with your application jar?

The method specifically that its trying to call on the client-side is a new one in CDH 5.12.0. However, given that its not finding that method yet referencing it, it appears that you may be bundling an older or newer (depending on what version your cluster runs) CDH/Hadoop dependencies with your application jar which is not a good idea (the platform provides these by default on its classpath).

Does an inbuilt example work normally? Try with:

hadoop org.apache.hadoop.examples.WordCount <in-dir> <out-dir>

P.s. Ensure you use and reference CDH dependencies in your project for Apache Hadoop (hadoop-client), not the upstream ones. The guide to do this is at https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo.html
Backline Customer Operations Engineer
Announcements