Put HiveQl NiFi processor is failing sometimes with kerberos issue

itsssudheer — Fri, 15 Jun 2018 03:36:37 GMT

It is running fine sometimes and failing at somepoint with the below error.

Put HiveQL Nifi processor is failing sometimes with Hive Connection pool - Error: Could not establish connection to jdbc:hive2://example.host.com:10001/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;: org.apache.http.client.ClientProtocolException (state=08S01,code=0)

Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
        at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
        at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:158)
        at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:150)
        at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:563)
        ... 35 common frames omitted
Caused by: org.apache.http.client.ClientProtocolException: null
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
        ... 41 common frames omitted
Caused by: org.apache.http.HttpException: null
        at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86)
        at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        ... 44 common frames omitted
Caused by: org.apache.http.HttpException: null
        at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68)
        at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
        ... 50 common frames omitted
Caused by: java.lang.reflect.UndeclaredThrowableException: null
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1887)
        at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
        at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
        ... 51 common frames omitted
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
        at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

itsssudheer — Fri, 15 Jun 2018 04:20:25 GMT

Yes it is nifi 1.5

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Shu_ashu — Fri, 15 Jun 2018 04:21:46 GMT

@Sudheer Velagapudi

Which version of NiFi/HDF are you using is it NiFi-1.5?

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Shu_ashu — Fri, 15 Jun 2018 04:47:18 GMT

@Sudheer Velagapudi

Once you disable/enable HiveController service the issue is not going to happen unitil sometime right?

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

itsssudheer — Fri, 15 Jun 2018 04:51:12 GMT

Exactly...

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Shu_ashu — Fri, 15 Jun 2018 04:57:28 GMT

@Sudheer Velagapudi

It's a bug in NiFi, couldn't able to refresh the kerberos ticket, to fix this issue please upgrade to HDF-3.1.2 (or) get in touch with support team regarding this issue to get Hot fix.

Refer to this link regarding HDF-3.1.2 release

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_release-notes/content/ch_hdf_relnotes.html

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

RahulSoni — Sat, 16 Jun 2018 02:16:17 GMT

@Sudheer Velagapudi

What version of HDF/NiFi are you using? As @Shu mentioned, this is a known issue with HDF 3.1.1. The reason is the Kerberos ticket is not auto-generated once it is expired. Due to which, the connection to Hive is not established and your query doesn't run.

I would recommend upgrading to HDF 3.1.2 to fix this issue, provided you are on a lower HDF version and this is the reason for your problem.

Please have a look at this document for further details on HDF 3.1.2 release and the issues that it is addressing.

question Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue in Archives of Support Questions (Read Only)

Put HiveQl NiFi processor is failing sometimes with kerberos issue

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue

Re: Put HiveQl NiFi processor is failing sometimes with kerberos issue