NiFi processor PutHiveQL cannot connect to Kerberized Hive

[#1]

Hello,

We cannot connect to Kerberized Hive (on HDP 2.5) from NiFi instance (NiFi 1.2.0); both PutHiveQL and PutHiveStreaming are erroring.

Using the same properties (principal and keytab values) and the same XML files (hive-site.xml, core-site.xml), NiFi can read and write from HDFS and HBase, so not sure why the problem only with Hive. Also, from the same NiFi server, using a simple java program, the connection to Hive works, it's only through NiFi that it fails.

Did a klist and it shows a valid ticket; also, manually ran kinit with principal and keytab and restarted NiFi, but still same error.

In nifi.properties, both nifi.kerberos.service.principal and nifi.kerberos.service.keytab.location are commented out, not sure if they should be uncommented or not, because those values are present in the processor properties; also, this entry is present in the properties file - nifi.kerberos.krb5.file=/etc/krb5.conf;

Below is the error trace from NiFi log for PutHiveQL:

2017-10-19 10:45:49,922 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.hive.PutHiveQL PutHiveQL[id=015a100e-2600-129d-7130-830324d7a86b] Failed to update Hive for StandardFlowFileRecord[uuid=320bc13f-b355-4b15-9518-2b3a03046d8f,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1508427927791-1, container=default, section=1], offset=0, length=87],offset=0,name=157442040917404,size=87] due to java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException; it is possible that retrying the operation will succeed, so routing to retry: java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
java.sql.SQLException: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:262)
	at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218)
	at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199)
	at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59)
	at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101)
	at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255)
	at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
	at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
	at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: org.apache.http.client.ClientProtocolException
	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
	at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
	at org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:219)
	at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:211)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
	... 23 common frames omitted
Caused by: org.apache.http.client.ClientProtocolException: null
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
	... 28 common frames omitted
Caused by: org.apache.http.HttpException: null
	at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86)
	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	... 31 common frames omitted
Caused by: org.apache.http.HttpException: null
	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68)
	at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
	... 37 common frames omitted
Caused by: java.lang.reflect.UndeclaredThrowableException: null
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
	... 38 common frames omitted
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
	... 40 common frames omitted

Any suggestions for what to check or what could be causing the issue ?

Thanks.

[#2]

@Raj B

Can you try to add, core-site.xml, hdfs-site.xml and hive-site.xml in the config resources properties ?

Please let us know the outcome ?

Thanks

[#3]

@njayakumar I already have those files included in the Hive Connection Pooling Service.

[#4]

@Raj B

based on your statement the same XML files (hive-site.xml, core-site.xml). Can you confirm the XML files ?

[#5]

if it is only hive-site.xml, core-site.xml. could you please add hdfs-site.xml ?

[#6]

@njayakumar, adding hdfs-site.xml didn't help.

[#7]

Also, uncommenting nifi.kerberos.service.principal and nifi.kerberos.service.keytab.location in nifi.properties didn't help either.

[#8]

@Raj B

What version of HDF/NiFi are you working with?

[#9]

@Wynner NiFi 1.2.0 and tried connecting to both HDP 2.5 and 2.6, neither works.

[#10]

If you are using Apache NiFi, the Hive processors are not compatible with HDP 2.5, because the Hive version on HDP is "newer" than the released versions of Apache Hive 1.2.x (the latter of which is used by Apache NiFi). In order to use the Hive processors with HDP, you should use the NiFi that comes with Hortonworks DataFlow (HDF). Version 3.0.x is based on NiFi 1.2.0 but is built with HDP Hive JARs and is compatible with HDP 2.5+.

[#11]

Hi @Matt Burgess,

AS per our latest comment, I am using Apache NiFi 1.4.0 and HDP 2.6. But we are still not able to connect to Hive from NiFi.

We tried adding libraries in the lib folder, the result of it is NiFi doesn't get launch.

Could you please help us?

Thank you.

Regards,

Khushboo Mehta.

,

Hi,

I am using Hortonworks NiFi 1.4.0 and HDP of v2.6. But still facing issues while routing a file from NiFi to Hive.

Thank you.

Regards,

Khushboo Mehta

[#12]

Is this current behavior similar to your other question? If so, I have left an answer there, I hope it helps resolve your issue.

[#13]

Hi , I am using latest hdf 3.x and nifi 1.2 and still getting the same error.

PutHiveQL is always throwing error something similar

cannot be started because it is not valid: '' is invalid because Failed to run validation due to java.lang.NullPointerException

nifi.processor.StandardValidationContext.isValidationRequired(StandardValidationContext.java)

I added 3 file location to the conf resources ( hdfs-site.xml, core-site.xml and hive-site.xml). Can anyone help me on this?

I am trying to load the data from hdfs to hive.

Step 1: GetFile

Step 2: PutHDFS

Step 3 : Replace Text . I added the hql in the replacement value ( create table nifi_test.clms (id int, name string) CLUSTERED BY(id) INTO 3 BUCKETS STORED AS ORC tblproperties("transactional"="true") LOCATION '${absolute.hdfs.path}';)

Step 4) PutHiveQL

All the above three steps are successful. Error is showing up in the Step 4.

If anyone provide one example, that would be great.