Created 02-14-2019 02:45 PM
I am following a tutorial of mysql to hive data suggested over https://community.hortonworks.com/articles/87686/rdbms-to-hive-using-nifi-small-medium-tables.html
issue is I am able to get data from mysql but unable to load it into hive . following is my stacktrace.
I am using standalone version of Apache Nifi 1.8.0 Hive 2.3.4 and hadoop 2.7.7 on my Mac machine. (I am not using any hortonwork or Cloudera virtual machine.) I installed Hadoop, hive and Nifi on my Mac/local machine.
initially it is able to connect with metastore then it loses the connection
2019-02-14 16:31:24,476 INFO [put-hive-streaming-0] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083
019-02-14 16:31:24,650 INFO [put-hive-streaming-0] hive.metastore Connected to metastore. 2019-02-14 16:31:24,650 INFO [Timer-Driven Process Thread-3] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083 2019-02-14 16:31:24,651 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore. 2019-02-14 16:31:25,148 WARN [Timer-Driven Process Thread-3] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException: null
org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:264)
2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Error connecting to Hive endpoint: table people3 at thrift://Bhushans-Mac-mini.local:9083
2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }:
Below is complete in details log :
2019-02-14 16:31:24,476 INFO [put-hive-streaming-0] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083
2019-02-14 16:31:24,650 INFO [put-hive-streaming-0] hive.metastore Connected to metastore.
2019-02-14 16:31:24,650 INFO [Timer-Driven Process Thread-3] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083
2019-02-14 16:31:24,651 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore.
2019-02-14 16:31:25,148 WARN [Timer-Driven Process Thread-3] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
at com.sun.proxy.$Proxy146.lock(Unknown Source)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547)
at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261)
at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73)
at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46)
at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:1139)
at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:1050)
at org.apache.nifi.processors.hive.PutHiveStreaming.access$1000(PutHiveStreaming.java:113)
at org.apache.nifi.processors.hive.PutHiveStreaming$1.lambda$process$2(PutHiveStreaming.java:843)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
at org.apache.nifi.processors.hive.PutHiveStreaming$1.process(PutHiveStreaming.java:840)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:793)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:659)
at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:659)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-02-14 16:31:26,152 INFO [Timer-Driven Process Thread-3] hive.metastore Trying to connect to metastore with URI thrift://Bhushans-Mac-mini.local:9083
2019-02-14 16:31:26,153 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore.
2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:79)
at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46)
at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:1139)
at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:1050)
at org.apache.nifi.processors.hive.PutHiveStreaming.access$1000(PutHiveStreaming.java:113)
at org.apache.nifi.processors.hive.PutHiveStreaming$1.lambda$process$2(PutHiveStreaming.java:843)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
at org.apache.nifi.processors.hive.PutHiveStreaming$1.process(PutHiveStreaming.java:840)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:793)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:659)
at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:659)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:264)
at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73)
... 24 common frames omitted
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547)
at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261)
... 25 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
at com.sun.proxy.$Proxy146.lock(Unknown Source)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573)
... 27 common frames omitted
2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Error connecting to Hive endpoint: table people3 at thrift://Bhushans-Mac-mini.local:9083
2019-02-14 16:31:26,157 ERROR [Timer-Driven Process Thread-3] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:631)
at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:649)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148)
at org.apache.nifi.processors.hive.PutHiveStreaming$1.process(PutHiveStreaming.java:840)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:793)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:659)
at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:659)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:79)
at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46)
at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:1139)
at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:1050)
at org.apache.nifi.processors.hive.PutHiveStreaming.access$1000(PutHiveStreaming.java:113)
at org.apache.nifi.processors.hive.PutHiveStreaming$1.lambda$process$2(PutHiveStreaming.java:843)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
... 18 common frames omitted
Caused by: org.apache.nifi.util.hive.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:264)
at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73)
... 24 common frames omitted
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://Bhushans-Mac-mini.local:9083', database='default', table='people3', partitionVals=[] }
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:547)
at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261)
... 25 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
at com.sun.proxy.$Proxy146.lock(Unknown Source)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573)
... 27 common frames omitted
2019-02-14 16:31:54,479 INFO [NiFi Web Server-1009] o.a.n.c.s.StandardProcessScheduler Stopping QueryDatabaseTable[id=e111ac04-0168-1000-22c0-2878ee5fb04e]
2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.QueryDatabaseTable
2019-02-14 16:31:54,480 INFO [Timer-Driven Process Thread-9] o.a.n.c.scheduling.QuartzSchedulingAgent Stopped scheduling QueryDatabaseTable[id=e111ac04-0168-1000-22c0-2878ee5fb04e] to run
2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.c.s.StandardProcessScheduler Stopping LogAttribute[id=e5c32b8e-0168-1000-b645-8a97dccea6a3]
2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.standard.LogAttribute
2019-02-14 16:31:54,480 INFO [Timer-Driven Process Thread-10] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling LogAttribute[id=e5c32b8e-0168-1000-b645-8a97dccea6a3] to run
2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.c.s.StandardProcessScheduler Stopping PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d]
2019-02-14 16:31:54,480 INFO [NiFi Web Server-1009] o.a.n.controller.StandardProcessorNode Stopping processor: class org.apache.nifi.processors.hive.PutHiveStreaming
2019-02-14 16:31:54,481 INFO [Timer-Driven Process Thread-6] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PutHiveStreaming[id=e11fcf36-0168-1000-8efc-f4294f04ef4d] to run
2019-02-14 16:31:54,801 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@29543345 // Another save pending = false
Created 02-14-2019 03:08 PM
The Hive processors in Apache NiFi work against Apache Hive 1.2.1, not Hive 2.x. Hive 2.x has difference in the Thrift interface from Hive 1.x, so you won't be able to use the NiFi Hive 1.2.x processors against any Hive 2.x instance (Apache or vendor). There may be work someday to support Hive 2.x but currently only Hive 1 and Hive 3 are supported in NiFi.
Created 02-15-2019 01:13 PM
Thanks Matt.. as per your guideline i tried using hive 1.2.2 and hive 3.1.1 both versions but still i am facing with same error. Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://172.52.100.33:9083', database='default', table='people3', partitionVals=[] }:
initially it is able to create connection with metastore but then it loses connection as per log below
2019-02-15 14:45:38,175 INFO [Timer-Driven Process Thread-3] hive.metastore Connected to metastore.
2019-02-15 14:45:38,543 WARN [Timer-Driven Process Thread-3] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
is there any guidance you can provide?
Created 02-15-2019 06:43 PM
It's possible (but would be quite unfortunate) that there was a Thrift change between 1.2.1 and 1.2.2, there were quite a few cases that went into 1.2.2.
Created 02-16-2019 04:17 AM
ok Matt... I will try again with Hive specific version of 1.2.1.
Hive 1.2.1+NiFi1.8+Hadoop 2.7.7 can this combination work or should i upgrade/downgrade any version? Or is there any virtualbox machine you can point me towards so that i can test mysql to hive flow using thrift server?
Created 03-03-2020 02:48 AM
I faced the same issue my nifi version 1.9.2 and hive 2.3.2 , is it resolved ?
Created 03-03-2020 01:25 PM
NiFi does not contain any processors that support Hive version 2.x. The latest versions of Apache NiFi offer Hive 1.x and Hive 3.x client based processor components. To support Hive 2.x version, you may need to build your own custom processors built using the Hive 2.x client.
Matt