NIFI doesn't start - shows "Too many open files" in nifi-app.logs

[#1]

I'm finding a situation where I'm unable to restart the NIFI instance. nifi-app.log error provided below. I noticed so many folders under content_repository/. The latest folder is named "1", I noticed 5 larger files in it. Is there anything I can change in NIFI property file to expire contents of the "content_repository", so that NIFI will start.

2018-05-25 06:57:12,805 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/190/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/190/archive: Too many open files
2018-05-25 06:57:12,805 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/191/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/191/archive: Too many open files
2018-05-25 06:57:12,806 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/192/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/192/archive: Too many open files

/content_repository has many sub-folders from 1-197, attached below latest few:
-------------------------------------------------------------------------------
drwxr-xr-x    3 user tp    21 May 11 12:37 8
drwxr-xr-x    3 user tp    21 May 11 12:37 6
drwxr-xr-x    3 user tp    21 May 11 12:37 5
drwxr-xr-x    3 user tp    21 May 11 12:37 4
drwxr-xr-x    3 user tp    21 May 11 12:37 3
drwxr-xr-x    3 user tp    21 May 11 12:37 7
drwxr-xr-x    3 user tp    21 May 14 06:55 9
drwxr-xr-x    3 user tp    21 May 14 06:55 10
drwxr-xr-x    3 user tp   136 May 24 06:55 1
drwxr-xr-x    3 user tp    44 May 24 08:39 2

Directory 1 - content:
----------------------
total 476
drwxr-xr-x    3 user    tp    136 May 24 06:55 .
drwxr-xr-x 1026 user     tp  20480 Jan 22 19:47 ..
-rw-r--r--    1 root     root    1637 May 15 15:52 1526411432690-1
-rw-r--r--    1 root     root   38369 May 17 14:32 1526558069151-1
-rw-r--r--    1 root     root   48687 May 22 11:45 1526990105222-1
-rw-r--r--    1 root     root     589 May 23 14:33 1527076500470-1
-rw-r--r--    1 root     root  341020 May 24 09:03 1527162955688-1 

drwxr-xr-x    2 user tpxes      6 May 22 06:55 archive

[#2 · Accepted solution]

@Winnie Philip

Check the current limit

cat /proc/sys/fs/file-max 

Run this command as root set it for example to 1million

# sysctl -w fs.file-max=1000000 

Then to permanently update it edit /etc/sysctl.conf and the below line

fs.file-max = 1000000

Retry and the "Too many open files" shouldn't appear

[#3]

Hi Geoffey, Thanks for the reply, The server has current limit of 3262161(ie. over 1M). I think I have an incorrect flow file in NIFI caused this large number of open file situation.

Can I create a new directory "content_repositoryX", and change below NIFI property to point the new directory and start NIFI? will it help to start NIFI?

nifi.content.repository.directory.default=./content_repositoryX

[#4]

Thank you, Geoffrey. This solution worked.

[#5]

Hoping you might be able to point me in the right direction.  If I'm experiencing this same error and "cat /proc/sys/fs/file-max" is reporting 3136547 "over 1 million" what would you suggest?  When I run "ls /proc/<PID>/fs/" on the Nifi "Process ID" it reports back a low and high limit of 4096.  Does this mean that my Nifi process is limited to 4096 open files at a given time?  I'm wondering whether the problem is that the limit is just to low or more of a Nifi issue that its failing to close files at a faster rate.