Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Mapreduce with phoenix in securedcluster fails with kerberos issue

Mapreduce with phoenix in securedcluster fails with kerberos issue

We are running a Mapreduce program which makes use of apache pheonix to get and process some data. The job fails with a kerberos error. The same program when run standalone runs fine. We are making use of keytab to authenticate and the keytab is used in the connection object to create to a phoenix connection. The MR job starts and launches containers,but the phoenix connection couldn't be established and the job fails with the error

2017-11-08 18:38:36,121 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Opening socket connection to server 172.16.3.197/172.16.3.197:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-08 18:38:36,122 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Socket connection established to 172.16.3.197/172.16.3.197:2181, initiating session
2017-11-08 18:38:36,138 INFO [main-SendThread(172.16.3.197:2181)] org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn: Session establishment complete on server 172.16.3.197/172.16.3.197:2181, sessionid = 0x35e92d03f140314, negotiated timeout = 40000
2017-11-08 18:38:36,693 INFO [main] org.apache.phoenix.metrics.Metrics: Initializing metrics system: phoenix
2017-11-08 18:38:36,693 WARN [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system already initialized!
2017-11-08 18:38:38,948 WARN [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2017-11-08 18:38:38,949 FATAL [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:612)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:157)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:738)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:735)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:735)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:897)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:866)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1209)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:217)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:318)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32831)
	at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:373)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:200)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:350)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:324)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
	... 26 more
2017-11-08 18:38:42,802 WARN [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2017-11-08 18:38:42,803 FATAL [hconnection-0xddf20fd-shared--pool1-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
3 REPLIES 3
Highlighted

Re: Mapreduce with phoenix in securedcluster fails with kerberos issue

Super Mentor

@ARUN

As the error indicates that there is no Valid Kerberos ticket hence the message is also suggesting to do a "kinit" to get a valid kerberos ticket:

org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.

.

So please do a "klist" to see if you have valid kerberos ticket or not? If not then please try doing the kinit On the correct KeyTab with Correct principal name.

.

Highlighted

Re: Mapreduce with phoenix in securedcluster fails with kerberos issue

Hi @Jay Kumar SenSharma, We have done a kinit at the start itself. Thats how the yarn containers are gtting launched. But from there the phoenix connection is not able to obtained

Highlighted

Re: Mapreduce with phoenix in securedcluster fails with kerberos issue

Super Collaborator

can you provide the code how you establish the phoenix connection?

Don't have an account?
Coming from Hortonworks? Activate your account here