Hi,   I have a CDH 5.1.3 cluster managed by cloudera manager. It's fully kerberised and uses AD 2003 as the KDC. I have enabled sentry and everything seems to work fine. However, every morning, beeline stops working and the hiveserver2 log shows the below issue. Currently the only fix i have is to restart hiveserver. With sentry disabled everything works fine. I've tried redeploying kerberos configs, regenerating credentials and increasing the kerberos ticket timeout. Anyone have any ideas where to look?   8:34:40.148 AM ERROR sentry.org.ape.thrift.transport.TSaslTransport SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at sentry.org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at sentry.org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253) at sentry.org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:1) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClient.java:115) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport.access$000(SentryPolicyServiceClient.java:77) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport$1.run(SentryPolicyServiceClient.java:101) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport$1.run(SentryPolicyServiceClient.java:99) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient$UgiSaslClientTransport.open(SentryPolicyServiceClient.java:99) at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient.<init>(SentryPolicyServiceClient.java:151) at org.apache.sentry.provider.db.SimpleDBProviderBackend.<init>(SimpleDBProviderBackend.java:52) at org.apache.sentry.provider.db.SimpleDBProviderBackend.<init>(SimpleDBProviderBackend.java:48) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.getAuthProvider(HiveAuthzBinding.java:247) at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.<init>(HiveAuthzBinding.java:88) at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.<init>(HiveAuthzBinding.java:81) at org.apache.sentry.binding.hive.HiveAuthzBindingHook.<init>(HiveAuthzBindingHook.java:98) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at java.lang.Class.newInstance(Class.java:374) at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:59) at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1162) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:440) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:352) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:995) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:988) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:98) at org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:163) at org.apache.hive.service.cli.session.HiveSessionImpl.runOperationWithLogCapture(HiveSessionImpl.java:514) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:222) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:204) at org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:168) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:316) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1373) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1358) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:57) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:244) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ... 49 more 8:34:40.151 AM WARN org.apache.hadoop.security.UserGroupInformation PriviledgedActionException as:hive/gb-slo-hdp-0001.dunnhumby.co.uk@DUNNHUMBY.CO.UK (auth:KERBEROS) cause:sentry.org.apache.thrift.transport.TTransportException: GSS initiate failed ... View more

I have a CDH 5.1.0 cluster, recently upgraded from CDH 5.0 via parcel. Since the upgrade I have been unable to start the NFS gateway service on any node. The error in the log is:   Exception in thread "main" java.io.IOException: Running in secure mode, but config doesn't have a keytab   I have found that there is no hdfs.keytab file being deployed to /var/run/cloudera-scm-agent/process/XXX-hdfs-NFSGATEWAY/ and the hdfs-site.xml is missing the dfs.nfs.keytab.file property. All other services are running fine. I can't find anything the configuration that seems out of place. The HDFS principal is ok as the Datanode process is working ok.   Anyone have any suggestions on how i can resolve this?   Thanks!   ... View more