NiFi Crash after runing chain of lookups

[#1]

Hi all,

My nifi crash for some reasons that I could not figure out, can anyone helps?

1. A chain of lookups that I have, the table being looked up contains around 50,000,000 rows

2. After running some node, it crashed like this:

3. After this, I relogged into NiFi to see what error happened:

My Lookup service setup:

I can provide more details upon request.

Please help!!!Thanks

[#2]

@jarviszzzz, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi @cotopaul @SAMSAL @TimothySpann experts who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

How many nodes?  How much RAM and CPUs per node?  Which version of NiFi?   Which version of JDK?

I recommend 128GB RAM, 32cores, 6 nodes+, JDK 21.

Also for that many rows you probably want to use a real database and not the tiny cache server.   I would say Redis could work.

[#4]

Dear Timothy,

I actually figured out the reason why it crash, so I change my local pc to a CPU server.

Below is the Pics for CPU and RAM:

I am actually using two server for this NiFi operation, one is holding Mysql Server, another one is for NiFi.

However, It still shows slowness for running the lookup processor. How can I make it faster? Thank you so much.

Please let me know if any other information that you need

[#5]

Hi Timothy,

Thank you so much for helping. I also made some changes (adding index, change data types etc.,) for mysql database. Now, it ran much faster than before