Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi putHIveProcessor multiple errors

Nifi putHIveProcessor multiple errors

New Contributor

I have a pretty standard flow to ingest data from kakfa topic and move it to hive table (transaction enabled and bucketed as well). processors used in flow are in sequence below

getkafka --> evaluateJSONPath --> replaceText(to build delimited file) --> mergeContent(10000recs)-->convertCSVtoAvro-->putHiveStreaming

The flow works fine until it hits a point and all transactions start queuing up with various errors like(and many more ignore the seq below)

  • metastore connection errors
  • Commit of Txn 317726473 failed on EndPoint
  • Unable to acquire lock on {metaStoreUri='thrift:.....
  • failed to process session due to java.lang.NullPointerException: java.lang.NullPointerException
  • o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect

The hivePutStreaming processor is then not responsive anymore. It take a lot of time to stop and restart but the transaction which have queued up so far get lost.

Nifi version - 1.1.0

HDP - 2.4.2.0.258

4 REPLIES 4

Re: Nifi putHIveProcessor multiple errors

New Contributor

Any help will be appreciated...

Thanks

Anand

Re: Nifi putHIveProcessor multiple errors

You may be running into HIVE-11934. Can you share more information about the NullPointerException? There should be a stack trace in logs/nifi-app.log.

Re: Nifi putHIveProcessor multiple errors

New Contributor

Hello Matt Burgess, thanks for getting back. Both me and Anand are colleagues investigating this issue.

This is the first warning, and then error that we get when stream. First it seems the connection is lost to the metastore (socket timeout exception), then there's an attempt to rollback the transactions and we see some errors for transactions already aborted/or even no such transaction errors.

2016-12-09 16:03:38,975 INFO [pool-1-thread-2] org.apache.nifi.io.socket.SocketListener Socket Listener has been terminated successfully.
2016-12-09 16:03:38,975 INFO [pool-1-thread-2] o.a.n.w.c.ApplicationStartupContextListener Flow service termination completed.
2016-12-09 16:03:38,975 INFO [pool-1-thread-2] /nifi-api Closing Spring root WebApplicationContext
2016-12-09 16:03:38,981 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Socket closed
java.net.SocketException: Socket closed
	at java.net.PlainSocketImpl.socketAccept(Native Method) ~[na:1.8.0_73]
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) ~[na:1.8.0_73]
	at java.net.ServerSocket.implAccept(ServerSocket.java:545) ~[na:1.8.0_73]
	at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348) ~[na:1.8.0_73]
	at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112) ~[nifi-socket-utils-1.1.0.jar:1.1.0]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
2016-12-09 16:03:38,984 ERROR [Timer-Driven Process Thread-2] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=97793ce3-bd20-1fdb-ae82-3402c810e9b1] PutHiveStreaming[id=97793ce3-bd20-1fdb-ae82-3402c810e9b1] failed to process due to org.apache.nifi.processor.exception.FlowFileAccessException: Could not flush OutputStream for StandardFlowFileRecord[uuid=c78fa934-4a61-429d-b683-2233ac100535,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1481299411098-14, container=default, section=14], offset=0, length=307870],offset=0,name=11171439018026104,size=307870]; rolling back session: org.apache.nifi.processor.exception.FlowFileAccessException: Could not flush OutputStream for StandardFlowFileRecord[uuid=c78fa934-4a61-429d-b683-2233ac100535,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1481299411098-14, container=default, section=14], offset=0, length=307870],offset=0,name=11171439018026104,size=307870] 

Then there is the unable to acquire lock, and the nullpointer exception and then the hive streaming becomes in an unrecoverable state from then onwards.

2016-12-09 16:33:41,700 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.hive.PutHiveStreaming 
org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://dev-master01.hdp-betsson.com:9083', database='edh', table='edh_logineventsnifibatch1', partitionVals=[2016-12-09] }: No such transaction txnid:317832007
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:684) ~[hive-hcatalog-streaming-1.2.1000.2.5.0.0-1245.jar:1.2.1000.2.5.0.0-1245]
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:651) ~[
hive-hcatalog-streaming-1.2.1000.2.5.0.0-1245.jar:1.2.1000.2.5.0.0-1245]
	at org.apache.nifi.util.hive.HiveWriter.nextTxn(HiveWriter.java:329) ~[nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.util.hive.HiveWriter.abort(HiveWriter.java:286) ~[nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.processors.hive.PutHiveStreaming.abortAllWriters(PutHiveStreaming.java:709) [nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.processors.hive.PutHiveStreaming.abortAndCloseWriters(PutHiveStreaming.java:696) [nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.jav
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_73]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_73]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_73]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_73]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_73]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73] 
2016-12-09 16:33:41,700 WARN [Timer-Driven Process Thread-7] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=97793ce3-bd20-1fdb-ae82-3402c810e9b1] unable to close hive connections. 
java.lang.NullPointerException: null
	at org.apache.nifi.util.hive.HiveWriter.toString(HiveWriter.java:92) ~[nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at java.lang.String.valueOf(String.java:2994) ~[na:1.8.0_73]
	at java.lang.StringBuilder.append(StringBuilder.java:131) ~[na:1.8.0_73]
	at org.apache.nifi.processors.hive.PutHiveStreaming.abortAllWriters(PutHiveStreaming.java:711) [nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.processors.hive.PutHiveStreaming.abortAndCloseWriters(PutHiveStreaming.java:696) [nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:584) [nifi-hive-processors-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) [nifi-api-1.1.0.jar:1.1.0]


Any suggestions?

Re: Nifi putHIveProcessor multiple errors

New Contributor

Anand Pandit are you able to resolve this issue? I am facing same issue for my project. I am using NiFI 1.8.0 hadoop 2.7.7 and hive 2.3.4. Can you tell me any solution you applied?

Don't have an account?
Coming from Hortonworks? Activate your account here