About mickt

mickt · ‎11-30-2021

Hey, we are running a Nifi cluster that connects to a Hive database to pull some data. After a period of time we get the below error. : java.net.SocketException: Broken pipe (Write failed).: java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed) Our current solution is to restart the Hive Controller services and this refresh's the connection and the process works again, this is obviously not ideal for a production implementation. We have also set the Kerberos user's timeout to be a large number of days in the hive-site.xml but the issue still persists. We are running Nifi version 1.12.1. Any help is very much appreciated.

mickt · ‎10-29-2021

Hey, Although the Nifi Registry persists the changes made to a process group, a Nifi cluster will do that regardless of having a registry or not. The Nifi Registry gives you the ability to create CI/CD across your pipelines. We have four separate Nifi clusters. Dev, Test, UAT & Production, all using the same Nifi Registry, when we make the changes in Dev the registry easily allows us to promote those changes to each of the environments with a full audit trail of who changed what and when. Critical for real life dataflows. Changes should only ever be made in dev and not directly in any of the other environments. Hope that helps

mickt · ‎08-13-2021

Hi Matt, Thank you for your response. If I am reading the issues correctly this fix will not be deployed until version 1.15 is released?

mickt · ‎08-11-2021

Hi, We have recently upgraded Nifi to 1.14 and are now receiving this error when trying to write to HDFS. "Failed to write to HDFS due to java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonIncludeProperties: java.lang.ClassNotFoundException: com.fasterxml.jackson.annotation.JsonIncludeProperties ↳ causes: java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonIncludeProperties" I have checked and all the jackson libraries are there. If anyone has any ideas it would be very much appreciated .

mickt · ‎02-22-2021

Hi turns out it was a permissions issue, so I had created a policy to allow the user to view the data provenance inside the Process Group but for that to work you need to add a global policy to allow that user to view the data provenance. Seems like a bit of a hack to me but thats what I had to do to get it working. Thanks for your help

mickt · ‎02-18-2021

Hi, I am having trouble getting the data provenance displaying in a clustered environment in Nifi. I have set all the required access policies and still the data provenance is empty. Any suggestions at this point is very much appreciated.

mickt · ‎02-12-2021

Hi, Could you provide a link to the Hortonworks hive nar file. The one provided in your answer seems to be broken. I am having a similar issue with NIFI and zookeeper and would like to test your solution out. Thank you in advance.

Online	Offline
Last Visited	‎10-19-2023 03:32 AM

Member Since	‎02-12-2021 01:22 AM
Last Visited	‎10-19-2023 03:32 AM
Posts	10

Cloudera Community

NIFI -> Hive Broke Pipe Error

Re: Using nifi registry in a nifi cluster.

Re: Nifi 1.14 PutHDFS Issue

Nifi 1.14 PutHDFS Issue

Re: Data Provenance not showing Nifi

Data Provenance not showing Nifi

Re: NiFi 1.10 Hive processors and backticks