Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Put HiveQl NiFi processor is failing sometimes with kerberos issue

Contributor

It is running fine sometimes and failing at somepoint with the below error.

Put HiveQL Nifi processor is failing sometimes with Hive Connection pool - Error: Could not establish connection to jdbc:hive2://example.host.com:10001/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;: org.apache.http.client.ClientProtocolException (state=08S01,code=0)

Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:158)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:150)
        at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:563)
        ... 35 common frames omitted
Caused by: org.apache.http.client.ClientProtocolException: null
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
        ... 41 common frames omitted
Caused by: org.apache.http.HttpException: null
        at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86)
        at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        ... 44 common frames omitted
Caused by: org.apache.http.HttpException: null
        at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68)
        at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
        ... 50 common frames omitted
Caused by: java.lang.reflect.UndeclaredThrowableException: null
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1887)
        at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
        at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
        ... 51 common frames omitted
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
        at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
1 ACCEPTED SOLUTION

Super Guru
@Sudheer Velagapudi

It's a bug in NiFi, couldn't able to refresh the kerberos ticket, to fix this issue please upgrade to HDF-3.1.2 (or) get in touch with support team regarding this issue to get Hot fix.

Refer to this link regarding HDF-3.1.2 release

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_release-notes/content/ch_hdf_relnotes.ht...

View solution in original post

6 REPLIES 6

Contributor

Yes it is nifi 1.5

Super Guru

@Sudheer Velagapudi

Once you disable/enable HiveController service the issue is not going to happen unitil sometime right?

Contributor

Exactly...

Super Guru

@Sudheer Velagapudi

Which version of NiFi/HDF are you using is it NiFi-1.5?

Super Guru
@Sudheer Velagapudi

It's a bug in NiFi, couldn't able to refresh the kerberos ticket, to fix this issue please upgrade to HDF-3.1.2 (or) get in touch with support team regarding this issue to get Hot fix.

Refer to this link regarding HDF-3.1.2 release

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_release-notes/content/ch_hdf_relnotes.ht...

@Sudheer Velagapudi

What version of HDF/NiFi are you using? As @Shu mentioned, this is a known issue with HDF 3.1.1. The reason is the Kerberos ticket is not auto-generated once it is expired. Due to which, the connection to Hive is not established and your query doesn't run.

I would recommend upgrading to HDF 3.1.2 to fix this issue, provided you are on a lower HDF version and this is the reason for your problem.

Please have a look at this document for further details on HDF 3.1.2 release and the issues that it is addressing.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.