Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datameer-Yarn2-Kerberos Integration issues

Datameer-Yarn2-Kerberos Integration issues

Contributor

Hello All,

 

I configured the cluste CDH 5.0.0 with kerberos and intergrated with Datameer all works  without kerberos but when I enable the kerberos, they job gets launced and finished but for some reason it is not giving below  error on datameer dashboard but the example PI runs from unix command prompt for datameer user id. 

 

Please give me some direction to move forward - kind of stuk for a week without able to move forward with YARN2/Kerberos/Datameer but same environment works with MRv1 with kerberos.

 

[anonymous]  INFO [2014-07-19 10:04:38.803] [MrPlanRunnerThread-0] (ClientServiceDelegate.java:273) - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
[anonymous]  INFO [2014-07-19 10:04:38.918] [MrPlanRunnerThread-0] (ClientServiceDelegate.java:273) - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
[anonymous]  INFO [2014-07-19 10:04:38.935] [pool-6-thread-1] (HdfsUploader.java:42) - Push job-log to hdfs://10.131.108.48:8020/user/datameer/joblogs/70/007-job.log
[anonymous]  WARN [2014-07-19 10:04:39.030] [MrPlanRunnerThread-0] (UserGroupInformation.java:1551) - PriviledgedActionException as:datameer/chapdc3hdpnm01.ops.tiaa-cref.org@HADOOP.TIAA-CREF.K5.ORG (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getTaskAttemptCompletionEvents(HistoryClientService.java:272)
	at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getTaskAttemptCompletionEvents(MRClientProtocolPBServiceImpl.java:173)
	at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:283)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

[anonymous]  INFO [2014-07-19 10:04:39.031] [MrPlanRunnerThread-0] (DelegateInputFormat.java:106) - Releasing splits (UUID: 82f442f9-096d-48b8-b16a-c4d958784eb8)  from cache, still cached split-arrays: 0
[anonymous]  INFO [2014-07-19 10:04:39.099] [MrPlanRunnerThread-0] (ClientServiceDelegate.java:273) - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
[anonymous]  INFO [2014-07-19 10:04:39.215] [MrPlanRunnerThread-0] (ClientServiceDelegate.java:273) - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
[anonymous]  INFO [2014-07-19 10:04:39.329] [MrPlanRunnerThread-0] (ClientServiceDelegate.java:273) - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
[anonymous]  WARN [2014-07-19 10:04:39.441] [MrPlanRunnerThread-0] (UserGroupInformation.java:1551) - PriviledgedActionException as:datameer/chapdc3hdpnm01.ops.tiaa-cref.org@HADOOP.TIAA-CREF.K5.ORG (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:223)
	at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
	at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
1 REPLY 1

Re: Datameer-Yarn2-Kerberos Integration issues

Explorer

Ram,

 

I also had this issue.  I believe Datameer is not pulling the correct value for yarn.app.mapreduce.am.staging-dir, as my staging jobs were being created under hdfs:/tmp/hadoop-yarn/staging/datameer, rather than /user/datameer.

 

Try adding the following to your Datameer Hadoop Cluster settings under Custom Properties:

 

yarn.app.mapreduce.am.staging-dir=/user

 

Another good clue I received from Datameer support was to make sure YARN Application Classpath matches the value from 'hadoop classpath' on your cluster.  Mine is:

 

$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/lib/*,$HADOOP_COMMON_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_YARN_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$HADOOP_MAPRED_HOME/*

 

Good luck,

 

Chad