Number of user processes for nifi user: Failed remote connection, input stream is closed.

[#1]

I found a ListSFTP processor with error "Failed remote connection, input stream is closed". This was temporarily resolved by increasing the nproc to 10240 from 1024.

Is this nproc count normal or should i be ready to increase the nproc to unlimited?

[#2 · Accepted solution]

@Joshua Adeleke

With NiFi dataflows it is typically to see large numbers of open file handles and user processes because of the concurrent thread operation supported by its many components.

In many cases you will find that the default ulimits for both open files and processes will fall short of what is needed for most dataflows.

I recommend setting these values to 50000 out the gate. You may find your self depending on volumes and complexity of your dataflow(s) needing to even set this higher. The default 1024 is almost always guaranteed to be an issue.

/etc/security/limits.conf

*          hard    nproc    50000
*          soft    nproc    50000
*          hard    nofiles  50000
*          soft    nofiles  50000

/etc/security/limits.d/90-nproc.conf

*          hard    nproc    50000
*          soft    nproc    50000

Thanks,

Matt

[#3]

Thank you @Matt Clarke.