[NIFI] Can't connect to HiveMetastore with putHiveStreaming processor (but putHiveQL works..)

[#1]

Hi,

I run NiFi on my windows machine and i try to put some data in Hive (HDP 2.6). I can connect to hive with a putHiveQL processor (so the network is ok between the two machcines), so with a jdbc connection but when i use the putHiveStreaming processor it doesn't work. I can't connect to Hive Metastore.

My configuration for the putHiveStreaming processor : (I added the link for my IP address in /etc/host)

The error :

2017-06-20 10:46:09,900 ERROR [Timer-Driven Process Thread-5] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=c4a50533-015c-1000-0a44-9c01c3cd6345] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@17777c40] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient: {}
org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@17777c40] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
 at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:535)
 at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54)
 at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:542)
 at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148)
 at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHiveStreaming.java:674)
 at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2125)
 at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2095)
 at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:628)
 at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:552)
 at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
 at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
 at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:552)
 at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
 at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
 at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
 at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
 at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
 at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227)
 at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202)
 at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157)
 at org.apache.nifi.util.hive.HiveWriter.lambda$newConnection$1(HiveWriter.java:246)
 at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:365)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 ... 3 common frames omitted
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
 at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
 at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
 at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
 at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118)
 at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230)
 at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227)
 at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
 at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
 at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
 ... 16 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
 at java.lang.reflect.Constructor.newInstance(Unknown Source)
 at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
 ... 26 common frames omitted
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect
 at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
 at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:420)
 at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)
 at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181)
 at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
 at java.lang.reflect.Constructor.newInstance(Unknown Source)
 at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
 at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
 at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
 at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:118)
 at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:230)
 at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:227)
 at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
 at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
 at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
 at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
 at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
 at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
 at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
 at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:227)
 at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:202)
 at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:448)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:274)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180)
 at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157)
 at org.apache.nifi.util.hive.HiveWriter.lambda$newConnection$1(HiveWriter.java:246)
 at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:365)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
 at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
 at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
 at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
 at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
 at java.net.PlainSocketImpl.connect(Unknown Source)
 at java.net.SocksSocketImpl.connect(Unknown Source)
 at java.net.Socket.connect(Unknown Source)
 at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
 ... 35 more


 at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:466)
 at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:236)
 at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:181)
 at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:330)
 ... 31 common frames omitted

Any idea ?

Thank you !

[#2]

Hi Quentin. Could you edit your question and provide the full stack trace that you will find in nifi-app.log? Thanks!

[#3]

Hi Pierre, it's done !

[#4]

@Quentin T,

It looks like you are using the sandbox, are you sure the port you are trying to reach is open on the sandbox? Can you check port forwarding if you are using the sandbox in a virtual machine? A connection refused error is very likely due to a network misconfiguration.

Hope this helps.

[#5]

Yes i'm using the sandbox (with virtualbox) but i set up a network bridge so it seems there is no port forwading..

[#6]

Can you telnet to that port on the sandbox from the host?

[#7]

Hi Matt,

No i can't..

And when i try #ss -an | grep 9083 on the sandbox-host i have nothing

[#8]

That is likely the source of your error. You will need to be able to access the Hive metastore port (which is 9083 by default) from the NiFi node.