Silver ⭐️⭐️⭐️

Question

NiFi Flow xml/json getting corrupted in multi node setup

11 replies
9893 views
Translate All

Hi All,

I have Nifi running on EKS cluser with 3 nodes. Every now and then, flow.xml.gz gets corrupted in one of the node and it fails to start back up again. Same things happens with other nodes sooner or later and ultimately NiFi crashes.

The error from nodes startup logs -

schema validation Error parsing Flow configuration at line <line_no>, col <col no > : cvc-complex-type:2.4.d : Invalid content was found starting with element 'property'. No child element is expected at this point.

It has become a pain to maintain Nifi because of this issue. Please suggest a solution.

Apache NiFi

VidyaSargur

Community Manager

@srv1009, Welcome to our community! To help you get the best possible answer, I have tagged our NiFi experts @MattWho @SAMSAL @cotopaul @DigitalPlumber who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Translate

Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.

srv1009Author

Silver ⭐️⭐️⭐️

I am doing Root cause Analysis, and source of issue seems to be abrupt nifi shutdown resulting in flow file corruption.
The logs before first error -

Received Trapped Signal.. Shutting down

Apache Nifi has accepted the shutdown signal and is shutting down now.

Failed to determine if process 165 is running or nor, assuming it is not.

Further information-

I am using 3 node nifi cluster, with 32GB memory. Heap memory allocated is 24GB.

Translate

cotopaul

Platinum ⭐️

@srv1009,

In my overall experience with NiFi, the problem you have reported only happened because of one of the following two reasons:
- as you already mentioned, an abrupt NiFi shutdown --> which of course might cause your corrupted file. This can be solved easily, you stop doing such shutdowns 🙂
- the physical disk on which NiFi is located gets corrupted and implicitly it starts corrupting other files as well. This has an easy solution as well --> you test your hardware health and if necessary you change it.

Nevertheless, the reason for why this happens is present either in the NiFi logs or the server logs 🙂

Translate

srv1009Author

Silver ⭐️⭐️⭐️

This shutdowns seems to be totally random as of now.
The logs which I have access to just says what I mentioned.
"Received trapped signal.. shutting down". I am trying to locate the source
of the issue but logs are not helping.

Could you tell me which log files to look into. I will try to get my hands
on them in lower environment and see if i can locate the source.

Translate

cotopaul

Platinum ⭐️

Assuming that you are running on Linux, you need to find your operating system logs. In most linux distribution, those logs are to be found in /var/log. Now, every sysadmin configures each server by your company rules and requirements so I suggest you speak with the team responsible for the linux server and ask them to provide you with the logs. In these logs, you might find out why you are receiving that error in the first place.

Unfortunately, this problem is not really related to NiFi, but to your infrastructure or to how somebody uses your NiFi instance. Somebody is doing something and you need to find out who and what 😞

Translate

srv1009Author

Silver ⭐️⭐️⭐️

Thanks for your response. It was very helpful and informative.

I am using Nifi 1.19 version and have tried using flow.json.gz and ran into same error there.

What I understand from ShutdownHook code is that whenever a SIGTERM signal is issued from k8s, it starts the shutdown and if it doesn't shutdown in the grace period configured in nifi.properties, the core process itself is forcefully destroyed.
As you said, the chances of getting flowfile corrupted is pretty slim as it needs many things to happen in succession. I was wondering what else could cause flowfile to go bad.
Also in the restart logs, I never saw error saying that flowfile has gone bad, it throws warning with schema errors and just shuts down. Only when i replace the flow xml or rename the default flow file to something else, the warning message disappears and nifi starts up.

Another information, I would like to bring in your notice is that it happens in our prod environments too, where flow is essentially read-only for everyone, so we are never making any change to the flow. Also, we have checkpointing enabled, every few minutes it archives the current flow state. Does it function same as making change to flow, i.e Archive the current flow xml and create new one? If yes, then it could fit in the rare sequence of events causing flow to corrupt.

Another thing,

Is it possible to configure nifi startup behaviour such that, if flow xml is corrupt, it ignores it and startup with new flow xml? As in Higher environments, losing the flow is not that big a concern, than to deploy nifi again with new flow xml.

we have nifi registry, our own flow deployment tools to sync the flow across environments to recover the flow quickly but nifi crashing every time something goes wrong with flow xml doesn't help to maintain High availability/resiliency.
Looking forward for your response.

Translate

MattWho

Titanium ⭐️⭐️⭐️

@srv1009

Are you sure you are removing or replacing a flow.xml.gz. Apache NiFi 1.19 uses the flow.json.gz.
The flow.json.gz is auto generated from the flow.xml.gz if and only if the flow.json.gz does not already exist at startup.

As far as flow changes in your production environment goes... All the following are flow config changes:
1. Moving a component
2. Changing state of a component (start, stop, enable, disable)
3. Importing a flow from the nifi-registry or changing the version of a version controlled flow.
So you are saying none of the above are happening ever in prod (seems unlikely)?

You mentioned that you have "checkpointing" enabled. The only "checkpointing" that NiFi does is NOT related the flow.ml.gz or the flow.json.gz. It has to do with checkpointing the flowfile_repository which contains metadata about NiFi FlowFiles queued within connection in your datflows on the canvas.

The flow.json.gz is only file being used on startup once it has been generated. Are you having disk space issues?
Archive can be enabled for the flow.json.gz as part of the following properties:

Archive of flow.json.gz only happens when a configuration change is made on the NiFi canvas or via some change via NiFi rest-api interaction resulting in changes to datafow(s). Archive when enabled does not randomly generate archive copies of the flow.json.gz. Only happens with each change. So if you are seeing an archive of teh flow.josn.gz generated every few minutes in the configured archive.dir? If so, then changes to your dataflow are happening.

Are you sure that the SIGTERM is executing a clean shutdown of the NiFi process (./nifi.sh stop) or is it just killing the NiFi process id? If a graceful shutdown is happening you would see that in the nifi-bootstrap.log.
The Graceful shutdown period is configurable by this property:

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Translate

srv1009Author

Silver ⭐️⭐️⭐️

Sorry for the late response. I wanted to come back with more information.
Yeah, I am sure that I replace flow.xml.gz to recover from this issue. As I mentioned, I have configured default flow file to flow.xml.gz using nifi properties, so it uses xml and as you said newer versions support json, I tried using json too.
Also I can confirm that shutdown is graceful. After much debugging and root cause analysis, I am able to reproduce the issue now and have the exact cause of error. Logs don't concur with my findings, but I seek your and community's help in confirming the issue and finding a resolution.

My Findings -

1) Nifi was failing for me in 1st restart itself.

2) From logs, its clear that It tries to Load flow , then runs probably DTD validation on it, throws warning and then shuts down.

3) These warnings comes with newly created basic flow(with couple of processors) as well, and new flow was able to load post warning, It moves ahead from the point where it throws warning.

4) If I put point 2 and 3 together, It was safe to logically assume that there is something in Flow itself causing nifi to fail on restart.

5) Everything in Flow was plain nifi processors doing bunch of tasks, However I had a ExecutScript block with python code to do some modifications on flowfile content. so, I created a fresh flow with just 3 processors - GenerateFlowFile, ExecuteScript, LogAttribute
In GenerateFlowfile I configured some random content. In ExecuteScript I copied code from https://community.cloudera.com/t5/Support-Questions/How-to-Iterate-the-Flow-file-in-Nifi-using-Python/m-p/242697 after indenting it properly and then just terminated the flow on LogAtrribute.
Basically a very simple flow to do some modificaion on flowfile content. That's it I kept them running , it was working fine. So, I went ahead and deleted the pod, logs indicated graceful shutdown. Pod restarted and got the exact same error, with warnings on schema and error server is shutting down.
I took fresh nifi instances and did it several times and it fails everytime with ExecuteScript block running python code.
Logs are not very informative but I can confirm that its the root cause of the issue. Can you or anyone else confirm if its replicable, if yes, what should be the resolution?

Nifi version used - 1.19.1

Translate

srv1009Author

Silver ⭐️⭐️⭐️

@cotopaul Please go through my latest response before concluding its a hardware or setup issue.
FYI, Nifi is working fine for more than a week for me since I removed the ExecuteScript Processor.

That's the only change I did and I replicated the issue several times before posting here.
Nifi has restarted several times since then without any issues.
could you please care to explain what sort of hardware issue it could be that it affects only ExecuteScript processor running python code?

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.