Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi - Too many open files error for PutCassandraQL

NiFi - Too many open files error for PutCassandraQL

New Contributor

Good day everyone, here are the details of my problem:

- NiFi 1.6

- A flow with 30 process groups

- Each group contains, ConsumeAMQP, MonitorActivity, JoltTransform, SplitJson, ReplaceText, ConvertRecord, PutMongoRecord, and PutCassandraQL processors

When the flow starts, I'm immediately locked out of the UI with "too many open files" errors - note that this happens when there is no data coming from the AMQP queues, which I assume means there are no flow files being created. I've followed the best practices to increase file handle limits just in case, but it's made no difference. Experimenting with using the API to stagger the start order of the processors has shown that the problem happens when all 30 PutCassandraQL processors are started. Has anyone ever encountered this issue?

Any help is appreciated, Thanks.

1 REPLY 1
Highlighted

Re: NiFi - Too many open files error for PutCassandraQL

@Nifemi Ogundare

Run the following command, as the user running NiFi, and post the output: ulimits -a

Also, the SplitJson processor could create a lot of files, depending on the data passing through.

Set the number of open file handles to 999999 and see if that helps.

Don't have an account?
Coming from Hortonworks? Activate your account here