Nifi Error on PutHiveStreaming

[#1]

2017-11-09T15:21:13,760 ERROR [pool-8-thread-29] metastore.RetryingHMSHandler: java.lang.IllegalStateException: Unexpected DataOperationType: UNSET agentInfo=Unknown txnid:601
        at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1000)
        at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:872)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6362)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy21.lock(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14155)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14139)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2017-11-09T15:21:13,760 INFO [pool-8-thread-29] org.apache.hadoop.hive.metastore.HiveMetaStore - 15: Done cleaning up thread local RawStore
2017-11-09T15:21:13,760 ERROR [pool-8-thread-29] server.TThreadPoolServer: Error occurred during processing of message.
java.lang.IllegalStateException: Unexpected DataOperationType: UNSET agentInfo=Unknown txnid:601
        at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1000) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:872) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6362) ~[hive-exec-2.3.0.jar:2.3.0]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) ~[hive-exec-2.3.0.jar:2.3.0]
        at com.sun.proxy.$Proxy21.lock(Unknown Source) ~[?:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14155) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14139) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) ~[hive-exec-2.3.0.jar:2.3.0]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_45]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_45]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) ~[hadoop-common-2.7.3.jar:?]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) ~[hive-exec-2.3.0.jar:2.3.0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[hive-exec-2.3.0.jar:2.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
2017-11-09T15:21:13,760  INFO [pool-8-thread-29] HiveMetaStore.audit: ugi=dwusr ip=127.0.0.1    cmd=Cleaning up thread local RawStore...
2017-11-09T15:21:13,760  INFO [pool-8-thread-29] HiveMetaStore.audit: ugi=dwusr ip=127.0.0.1    cmd=Done cleaning up thread local RawStore
2017-11-09T15:21:14,783 ERROR [pool-8-thread-32] metastore.RetryingHMSHandler: java.lang.IllegalStateException: Unexpected DataOperationType: UNSET agentInfo=Unknown txnid:601
        at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1000)
        at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:872)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6362)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
        at com.sun.proxy.$Proxy21.lock(Unknown Source)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14155)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14139)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

I am getting above error when running PutHiveStreaming processor in Nifi

Any idea what's wrong here.

I am using Hive 2.3.0 and Nifi version is 1.4.0

[#2 · Accepted solution]

The Hive processors in Apache NiFi 1.4.0 are built against Apache NiFi 1.2.1, so are not guaranteed to work with Apache Hive 2.3.0. If you are using the HDP platform, it has a Hive version based on 1.2.x but is closer to 2.0. Apache NiFi's Hive processors are not compatible with HDP 2.5+, you will likely want to use the NiFi-only Hortonworks Data Flow (HDF) package. This version of NiFi is built against the HDP Hive 1.2.x version. Having said that, HDF NiFi might work for your case, but it is also not guaranteed / supported to work against Hive 2.3.0 (whether Apache Hive or HDP Hive), as the baseline is still 1.2.x. The currently supported configuration is HDF NiFi against HDP Hive 1.2.x.

[#3]

@Matt Burgess

My Nifi Hive NAR file is nifi-hive-nar-1.4.0.nar . Could you please tell me which NAR version i should be using to make it work.

[#4]

That's the thing, I'm not sure any NAR version will work. Are you using Apache Hive 2.3.0 or a vendor-specific version?

[#5]

I am using Apache Hive 2.3.0. Please recommend me the HDP version i should go for. I think manual setup won;t work for me.

[#6]

@Matt Burgess , this issue was resolved by downloading HDF version of Nifi 1.2.0.. Thanks