Reply
Expert Contributor
Posts: 131
Registered: ‎08-08-2013

"hadoop queue" cmd exits with NullPointer exception

Hi,

 

I configured YARN to use the CapacityScheduler and wanted to check the queues and the acl's afterwards on the shell.

The RM-WebUI shows the configured queues (root, emergency, lowprio) as expected, but if I execute the shell command

 

sudo -u yarn hadoop queue -list

 

I get the following exception:

 

DEPRECATED: Use of this script to execute mapred command is deprecated.
Instead use the mapred command for it.

14/05/05 12:11:11 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/05/05 12:11:11 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.mapred.JobClient.getJobQueueInfoArray(JobClient.java:1045)
at org.apache.hadoop.mapred.JobClient.access$1800(JobClient.java:139)
at org.apache.hadoop.mapred.JobClient$10.run(JobClient.java:1063)
at org.apache.hadoop.mapred.JobClient$10.run(JobClient.java:1061)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.JobClient.getRootQueues(JobClient.java:1061)
at org.apache.hadoop.mapred.JobQueueClient.displayQueueList(JobQueueClient.java:148)
at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:101)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:240)

 

It doesn't matter if I execute the cmd as user 'yarn' or another one, the exception is the same. As mentioned earlier, the queues are looking fine in the ResourceManager-WebUI.

 

What's the problem on the shell execution ?!?!

Highlighted
Posts: 1,894
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: "hadoop queue" cmd exits with NullPointer exception

This will happen if your host lacks client configurations, which causes the default mapreduce.framework.name value of "local" to be used instead of a connection to your actual RM.

Here's how you can manually reproduce it even if you have a working command already:

➜ ~ hadoop queue -Dmapreduce.framework.name=local -list
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.mapred.JobClient.getJobQueueInfoArray(JobClient.java:1045)
at org.apache.hadoop.mapred.JobClient.access$1800(JobClient.java:139)
at org.apache.hadoop.mapred.JobClient$10.run(JobClient.java:1063)
at org.apache.hadoop.mapred.JobClient$10.run(JobClient.java:1061)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.JobClient.getRootQueues(JobClient.java:1061)
at org.apache.hadoop.mapred.JobQueueClient.displayQueueList(JobQueueClient.java:148)
at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:101)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:240)

If you have proper client configs including mapred-site.xml and yarn-site.xml that correctly define the cluster information, then the command will work properly and list queues:

➜ ~ hadoop queue -list
14/07/20 18:15:24 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
======================
Queue Name : root.default
Queue State : running
Scheduling Info : Capacity: 0.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0
Announcements