Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NIFI doesn't start - shows "Too many open files" in nifi-app.logs

avatar
Contributor

I'm finding a situation where I'm unable to restart the NIFI instance. nifi-app.log error provided below. I noticed so many folders under content_repository/. The latest folder is named "1", I noticed 5 larger files in it. Is there anything I can change in NIFI property file to expire contents of the "content_repository", so that NIFI will start.

2018-05-25 06:57:12,805 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/190/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/190/archive: Too many open files
2018-05-25 06:57:12,805 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/191/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/191/archive: Too many open files
2018-05-25 06:57:12,806 WARN [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository Failed to cleanup archived files in /appl/nifi/nifi-1.5.0/content_repository/192/archive due to java.nio.file.FileSystemException: /appl/nifi/nifi-1.5.0/content_repository/192/archive: Too many open files

/content_repository has many sub-folders from 1-197, attached below latest few:
-------------------------------------------------------------------------------
drwxr-xr-x    3 user tp    21 May 11 12:37 8
drwxr-xr-x    3 user tp    21 May 11 12:37 6
drwxr-xr-x    3 user tp    21 May 11 12:37 5
drwxr-xr-x    3 user tp    21 May 11 12:37 4
drwxr-xr-x    3 user tp    21 May 11 12:37 3
drwxr-xr-x    3 user tp    21 May 11 12:37 7
drwxr-xr-x    3 user tp    21 May 14 06:55 9
drwxr-xr-x    3 user tp    21 May 14 06:55 10
drwxr-xr-x    3 user tp   136 May 24 06:55 1
drwxr-xr-x    3 user tp    44 May 24 08:39 2

Directory 1 - content:
----------------------
total 476
drwxr-xr-x    3 user    tp    136 May 24 06:55 .
drwxr-xr-x 1026 user     tp  20480 Jan 22 19:47 ..
-rw-r--r--    1 root     root    1637 May 15 15:52 1526411432690-1
-rw-r--r--    1 root     root   38369 May 17 14:32 1526558069151-1
-rw-r--r--    1 root     root   48687 May 22 11:45 1526990105222-1
-rw-r--r--    1 root     root     589 May 23 14:33 1527076500470-1
-rw-r--r--    1 root     root  341020 May 24 09:03 1527162955688-1 

drwxr-xr-x    2 user tpxes      6 May 22 06:55 archive
1 ACCEPTED SOLUTION

avatar
Master Mentor

@Winnie Philip

Check the current limit

cat /proc/sys/fs/file-max 

Run this command as root set it for example to 1million

# sysctl -w fs.file-max=1000000 

Then to permanently update it edit /etc/sysctl.conf and the below line

fs.file-max = 1000000

Retry and the "Too many open files" shouldn't appear

View solution in original post

4 REPLIES 4

avatar
Master Mentor

@Winnie Philip

Check the current limit

cat /proc/sys/fs/file-max 

Run this command as root set it for example to 1million

# sysctl -w fs.file-max=1000000 

Then to permanently update it edit /etc/sysctl.conf and the below line

fs.file-max = 1000000

Retry and the "Too many open files" shouldn't appear

avatar
Contributor

Hi Geoffey, Thanks for the reply, The server has current limit of 3262161(ie. over 1M). I think I have an incorrect flow file in NIFI caused this large number of open file situation.

Can I create a new directory "content_repositoryX", and change below NIFI property to point the new directory and start NIFI? will it help to start NIFI?

nifi.content.repository.directory.default=./content_repositoryX

avatar
Contributor

Thank you, Geoffrey. This solution worked.

avatar
New Contributor

Hoping you might be able to point me in the right direction.  If I'm experiencing this same error and "cat /proc/sys/fs/file-max" is reporting 3136547 "over 1 million" what would you suggest?  When I run "ls /proc/<PID>/fs/" on the Nifi "Process ID" it reports back a low and high limit of 4096.  Does this mean that my Nifi process is limited to 4096 open files at a given time?  I'm wondering whether the problem is that the limit is just to low or more of a Nifi issue that its failing to close files at a faster rate.