We have an issue where Nifi crashes every few days. By crashing, I mean that it is up and running on a Friday afternoon, but when I return on Monday and check its status it is down. Getting it back online is not an issue, it starts right up after running the appropriate command.
The last messages in the nifi-app.log have a series of this WARN message:
[Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in ..../repos/content_repository/.../archive due to java.nio.file.FileSystemException: ..../repos/content_repository/..../archive: Too many open files
and ends with a single ERROR message:
[Timer-Driven Process Thread-9] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[...] Routing to Failure due to exception: java.net.SocketException: Too many open files: java.net.SocketException: Too many open files
The output for "cat /proc/sys/fs/file-max" is reporting 3136547 "over 1 million".
The output for "ls /proc/<PID>/fs/" on the Nifi "Process ID" reports back a low and high limit of 4096.
Does this mean that my Nifi process is limited to 4096 open files at a given time? I'm wondering whether the problem is that the limit is just too low or more of a Nifi issue that its failing to close files at a fast enough rate.