question Re: NIFI doesn't start - shows "Too many open files" in nifi-app.logs in Archives of Support Questions (Read Only)

NIFI doesn't start - shows "Too many open files" in nifi-app.logs

winnie_philip — Fri, 25 May 2018 22:53:41 GMT

I'm finding a situation where I'm unable to restart the NIFI instance. nifi-app.log error provided below. I noticed so many folders under content_repository/. The latest folder is named "1", I noticed 5 larger files in it. Is there anything I can change in NIFI property file to expire contents of the "content_repository", so that NIFI will start.

2018-05-25 06:57:12,805 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/190/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/190/archive: Too many open files
2018-05-25 06:57:12,805 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/191/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/191/archive: Too many open files
2018-05-25 06:57:12,806 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/192/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/192/archive: Too many open files

/content_repository has many sub-folders from 1-197, attached below latest few:
-------------------------------------------------------------------------------
drwxr-xr-x    3 user tp    21 May 11 12:37 8
drwxr-xr-x    3 user tp    21 May 11 12:37 6
drwxr-xr-x    3 user tp    21 May 11 12:37 5
drwxr-xr-x    3 user tp    21 May 11 12:37 4
drwxr-xr-x    3 user tp    21 May 11 12:37 3
drwxr-xr-x    3 user tp    21 May 11 12:37 7
drwxr-xr-x    3 user tp    21 May 14 06:55 9
drwxr-xr-x    3 user tp    21 May 14 06:55 10
drwxr-xr-x    3 user tp   136 May 24 06:55 1
drwxr-xr-x    3 user tp    44 May 24 08:39 2

Directory 1 - content:
----------------------
total 476
drwxr-xr-x    3 user    tp    136 May 24 06:55 .
drwxr-xr-x 1026 user     tp  20480 Jan 22 19:47 ..
-rw-r--r--    1 root     root    1637 May 15 15:52 1526411432690-1
-rw-r--r--    1 root     root   38369 May 17 14:32 1526558069151-1
-rw-r--r--    1 root     root   48687 May 22 11:45 1526990105222-1
-rw-r--r--    1 root     root     589 May 23 14:33 1527076500470-1
-rw-r--r--    1 root     root  341020 May 24 09:03 1527162955688-1 

drwxr-xr-x    2 user tpxes      6 May 22 06:55 archive

Re: NIFI doesn't start - shows "Too many open files" in nifi-app.logs

Shelton — Fri, 25 May 2018 23:08:30 GMT

@Winnie Philip

Check the current limit

cat /proc/sys/fs/file-max

Run this command as root set it for example to 1million

# sysctl -w fs.file-max=1000000

Then to permanently update it edit /etc/sysctl.conf and the below line

fs.file-max = 1000000

Retry and the "Too many open files" shouldn't appear

Re: NIFI doesn't start - shows "Too many open files" in nifi-app.logs

winnie_philip — Fri, 25 May 2018 23:21:02 GMT

Hi Geoffey, Thanks for the reply, The server has current limit of 3262161(ie. over 1M). I think I have an incorrect flow file in NIFI caused this large number of open file situation.

Can I create a new directory "content_repositoryX", and change below NIFI property to point the new directory and start NIFI? will it help to start NIFI?

nifi.content.repository.directory.default=./content_repositoryX

Re: NIFI doesn't start - shows "Too many open files" in nifi-app.logs

winnie_philip — Sat, 02 Jun 2018 08:58:32 GMT

Thank you, Geoffrey. This solution worked.

Re: NIFI doesn't start - shows "Too many open files" in nifi-app.logs

wg21908 — Mon, 15 Mar 2021 14:34:11 GMT

Hoping you might be able to point me in the right direction. If I'm experiencing this same error and "cat /proc/sys/fs/file-max" is reporting 3136547 "over 1 million" what would you suggest? When I run "ls /proc/<PID>/fs/" on the Nifi "Process ID" it reports back a low and high limit of 4096. Does this mean that my Nifi process is limited to 4096 open files at a given time? I'm wondering whether the problem is that the limit is just to low or more of a Nifi issue that its failing to close files at a faster rate.