Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi processor PutHiveQL cannot connect to Kerberized Hive

NiFi processor PutHiveQL cannot connect to Kerberized Hive

Rising Star

Hello,

We cannot connect to Kerberized Hive (on HDP 2.5) from NiFi instance (NiFi 1.2.0); both PutHiveQL and PutHiveStreaming are erroring.

Using the same properties (principal and keytab values) and the same XML files (hive-site.xml, core-site.xml), NiFi can read and write from HDFS and HBase, so not sure why the problem only with Hive. Also, from the same NiFi server, using a simple java program, the connection to Hive works, it's only through NiFi that it fails.

Did a klist and it shows a valid ticket; also, manually ran kinit with principal and keytab and restarted NiFi, but still same error.

In nifi.properties, both nifi.kerberos.service.principal and nifi.kerberos.service.keytab.location are commented out, not sure if they should be uncommented or not, because those values are present in the processor properties; also, this entry is present in the properties file - nifi.kerberos.krb5.file=/etc/krb5.conf;

Below is the error trace from NiFi log for PutHiveQL:

2017-10-19 10:45:49,922 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.hive.PutHiveQL PutHiveQL[id=015a100e-2600-129d-7130-830324d7a86b] Failed to update Hive for StandardFlowFileRecord[uuid=320bc13f-b355-4b15-9518-2b3a03046d8f,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1508427927791-1, container=default, section=1], offset=0, length=87],offset=0,name=157442040917404,size=87] due to java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException; it is possible that retrying the operation will succeed, so routing to retry: java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:262)
	at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218)
	at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199)
	at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59)
	at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255)
	at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
	at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
	at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
	at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
	at org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:219)
	at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:211)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
	... 23 common frames omitted
Caused by: org.apache.http.client.ClientProtocolException: null
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
	... 28 common frames omitted
Caused by: org.apache.http.HttpException: null
	at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86)
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	... 31 common frames omitted
Caused by: org.apache.http.HttpException: null
	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68)
	at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
	... 37 common frames omitted
Caused by: java.lang.reflect.UndeclaredThrowableException: null
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
	... 38 common frames omitted
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
	... 40 common frames omitted

Any suggestions for what to check or what could be causing the issue ?

Thanks.

12 REPLIES 12

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

Contributor

@Raj B

Can you try to add, core-site.xml, hdfs-site.xml and hive-site.xml in the config resources properties ?

Please let us know the outcome ?

Thanks

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

Rising Star

@njayakumar I already have those files included in the Hive Connection Pooling Service.

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

Contributor

@Raj B

based on your statement the same XML files (hive-site.xml, core-site.xml). Can you confirm the XML files ?

Highlighted

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

Contributor

if it is only hive-site.xml, core-site.xml. could you please add hdfs-site.xml ?

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

Rising Star

@njayakumar, adding hdfs-site.xml didn't help.

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

Rising Star

Also, uncommenting nifi.kerberos.service.principal and nifi.kerberos.service.keytab.location in nifi.properties didn't help either.

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

@Raj B

What version of HDF/NiFi are you working with?

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

Rising Star

@Wynner NiFi 1.2.0 and tried connecting to both HDP 2.5 and 2.6, neither works.

Re: NiFi processor PutHiveQL cannot connect to Kerberized Hive

If you are using Apache NiFi, the Hive processors are not compatible with HDP 2.5, because the Hive version on HDP is "newer" than the released versions of Apache Hive 1.2.x (the latter of which is used by Apache NiFi). In order to use the Hive processors with HDP, you should use the NiFi that comes with Hortonworks DataFlow (HDF). Version 3.0.x is based on NiFi 1.2.0 but is built with HDP Hive JARs and is compatible with HDP 2.5+.