Created 06-14-2018 08:36 PM
It is running fine sometimes and failing at somepoint with the below error.
Put HiveQL Nifi processor is failing sometimes with Hive Connection pool - Error: Could not establish connection to jdbc:hive2://example.host.com:10001/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;: org.apache.http.client.ClientProtocolException (state=08S01,code=0)
Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297) at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at org.apache.hive.service.cli.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:158) at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:150) at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:563) ... 35 common frames omitted Caused by: org.apache.http.client.ClientProtocolException: null at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251) ... 41 common frames omitted Caused by: org.apache.http.HttpException: null at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86) at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ... 44 common frames omitted Caused by: org.apache.http.HttpException: null at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68) at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74) ... 50 common frames omitted Caused by: java.lang.reflect.UndeclaredThrowableException: null at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1887) at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83) at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62) ... 51 common frames omitted Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183) at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
Created 06-14-2018 09:57 PM
It's a bug in NiFi, couldn't able to refresh the kerberos ticket, to fix this issue please upgrade to HDF-3.1.2 (or) get in touch with support team regarding this issue to get Hot fix.
Refer to this link regarding HDF-3.1.2 release
Created 06-14-2018 09:20 PM
Yes it is nifi 1.5
Created 06-14-2018 09:47 PM
Once you disable/enable HiveController service the issue is not going to happen unitil sometime right?
Created 06-14-2018 09:51 PM
Exactly...
Created 06-14-2018 09:21 PM
Which version of NiFi/HDF are you using is it NiFi-1.5?
Created 06-14-2018 09:57 PM
It's a bug in NiFi, couldn't able to refresh the kerberos ticket, to fix this issue please upgrade to HDF-3.1.2 (or) get in touch with support team regarding this issue to get Hot fix.
Refer to this link regarding HDF-3.1.2 release
Created 06-15-2018 07:16 PM
What version of HDF/NiFi are you using? As @Shu mentioned, this is a known issue with HDF 3.1.1. The reason is the Kerberos ticket is not auto-generated once it is expired. Due to which, the connection to Hive is not established and your query doesn't run.
I would recommend upgrading to HDF 3.1.2 to fix this issue, provided you are on a lower HDF version and this is the reason for your problem.
Please have a look at this document for further details on HDF 3.1.2 release and the issues that it is addressing.