Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi PutHiveStraming issue

avatar

I am following a tutorial of mysql to hive data suggested over https://community.hortonworks.com/articles/87686/rdbms-to-hive-using-nifi-small-medium-tables.html
issue is I am able to get data from mysql but unable to load it into hive . following is my stacktrace.
I am using standalone version of Apache Nifi 1.8.0 Hive 2.3.4 and hadoop 2.7.7 on my Mac machine. (I am not using any hortonwork or Cloudera virtual machine.) I installed Hadoop, hive and Nifi on my Mac/local machine.

initially it is able to connect with metastore then it loses the connection

2019-02-14 16:31:24,476 INFO [put-hive-streaming-0] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083

019-02-14 16:31:24,650 INFO [put-hive-streaming-0] hive.metastore Connected to metastore. 2019-02-14 16:31:24,650 INFO [Timer-Driven Process Thread-3] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083 2019-02-14 16:31:24,651 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore. 2019-02-14 16:31:25,148 WARN [Timer-Driven Process Thread-3] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException: null

org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:264)

2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Error connecting to Hive endpoint: table people3 at thrift://Bhushans-Mac-mini.local:9083 2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }:

Below is complete in details log :
2019-02-14 16:31:24,476 INFO [put-hive-streaming-0] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083 2019-02-14 16:31:24,650 INFO [put-hive-streaming-0] hive.metastore Connected to metastore. 2019-02-14 16:31:24,650 INFO [Timer-Driven Process Thread-3] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083 2019-02-14 16:31:24,651 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore. 2019-02-14 16:31:25,148 WARN [Timer-Driven Process Thread-3] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152) at com.sun.proxy.$Proxy146.lock(Unknown Source) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547) at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261) at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73) at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46) at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:1139) at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:1050) at org.apache.nifi.processors.hive.PutHiveStreaming.access$1000(PutHiveStreaming.java:113) at org.apache.nifi.processors.hive.PutHiveStreaming$1.lambda$process$2(PutHiveStreaming.java:843) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) at org.apache.nifi.processors.hive.PutHiveStreaming$1.process(PutHiveStreaming.java:840) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:793) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:659) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:659) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-02-14 16:31:26,152 INFO [Timer-Driven Process Thread-3] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083 2019-02-14 16:31:26,153 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore. 2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:79) at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46) at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:1139) at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:1050) at org.apache.nifi.processors.hive.PutHiveStreaming.access$1000(PutHiveStreaming.java:113) at org.apache.nifi.processors.hive.PutHiveStreaming$1.lambda$process$2(PutHiveStreaming.java:843) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) at org.apache.nifi.processors.hive.PutHiveStreaming$1.process(PutHiveStreaming.java:840) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:793) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:659) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:659) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:264) at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73) ... 24 common frames omitted Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547) at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261) ... 25 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152) at com.sun.proxy.$Proxy146.lock(Unknown Source) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573) ... 27 common frames omitted 2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Error connecting to Hive endpoint: table people3 at thrift://Bhushans-Mac-mini.local:9083 2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:631) at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:649) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148) at org.apache.nifi.processors.hive.PutHiveStreaming$1.process(PutHiveStreaming.java:840) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:793) at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:659) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:659) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:79) at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46) at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:1139) at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:1050) at org.apache.nifi.processors.hive.PutHiveStreaming.access$1000(PutHiveStreaming.java:113) at org.apache.nifi.processors.hive.PutHiveStreaming$1.lambda$process$2(PutHiveStreaming.java:843) at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127) ... 18 common frames omitted Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:264) at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73) ... 24 common frames omitted Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] } at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547) at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261) ... 25 common frames omitted Caused by: org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152) at com.sun.proxy.$Proxy146.lock(Unknown Source) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573) ... 27 common frames omitted 2019-02-14 16:31:54,479 INFO [NiFi Web Server-1009] o.a.n.c.s.StandardProcessScheduler Stopping QueryDatabaseTable[id=e111ac04-0168-1000-22c0-2878ee5fb04e] 2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.QueryDatabaseTable 2019-02-14 16:31:54,480 INFO [Timer-Driven Process Thread-9] o.a.n.c.scheduling.QuartzSchedulingAgent Stopped scheduling QueryDatabaseTable[id=e111ac04-0168-1000-22c0-2878ee5fb04e] to run 2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.c.s.StandardProcessScheduler Stopping LogAttribute[id=e5c32b8e-0168-1000-b645-8a97dccea6a3] 2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.LogAttribute 2019-02-14 16:31:54,480 INFO [Timer-Driven Process Thread-10] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling LogAttribute[id=e5c32b8e-0168-1000-b645-8a97dccea6a3] to run 2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.c.s.StandardProcessScheduler Stopping PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] 2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.hive.PutHiveStreaming 2019-02-14 16:31:54,481 INFO [Timer-Driven Process Thread-6] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] to run 2019-02-14 16:31:54,801 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@29543345 // Another save pending = false

6 REPLIES 6

avatar
Master Guru

The Hive processors in Apache NiFi work against Apache Hive 1.2.1, not Hive 2.x. Hive 2.x has difference in the Thrift interface from Hive 1.x, so you won't be able to use the NiFi Hive 1.2.x processors against any Hive 2.x instance (Apache or vendor). There may be work someday to support Hive 2.x but currently only Hive 1 and Hive 3 are supported in NiFi.

avatar

Thanks Matt.. as per your guideline i tried using hive 1.2.2 and hive 3.1.1 both versions but still i am facing with same error. Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://172.52.100.33:9083', database='default', table='people3', partitionVals=[] }:
initially it is able to create connection with metastore but then it loses connection as per log below
2019-02-15 14:45:38,175 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore. 2019-02-15 14:45:38,543 WARN [Timer-Driven Process Thread-3] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
is there any guidance you can provide?

avatar
Master Guru

It's possible (but would be quite unfortunate) that there was a Thrift change between 1.2.1 and 1.2.2, there were quite a few cases that went into 1.2.2.

avatar

ok Matt... I will try again with Hive specific version of 1.2.1.
Hive 1.2.1+NiFi1.8+Hadoop 2.7.7 can this combination work or should i upgrade/downgrade any version? Or is there any virtualbox machine you can point me towards so that i can test mysql to hive flow using thrift server?

avatar
New Contributor

I faced the same issue my nifi version 1.9.2 and hive 2.3.2 , is it resolved ?

avatar
Super Mentor

@asfou 

 

NiFi does not contain any processors that support Hive version 2.x.  The latest versions of Apache NiFi offer Hive 1.x and Hive 3.x client based processor components.  To support Hive 2.x version, you may need to build your own custom processors built using the Hive 2.x client.

Matt