Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Number of user processes for nifi user: Failed remote connection, input stream is closed.

avatar
Expert Contributor

I found a ListSFTP processor with error "Failed remote connection, input stream is closed". This was temporarily resolved by increasing the nproc to 10240 from 1024.

Is this nproc count normal or should i be ready to increase the nproc to unlimited?

1 ACCEPTED SOLUTION

avatar
Super Mentor
@Joshua Adeleke

With NiFi dataflows it is typically to see large numbers of open file handles and user processes because of the concurrent thread operation supported by its many components.

In many cases you will find that the default ulimits for both open files and processes will fall short of what is needed for most dataflows.

I recommend setting these values to 50000 out the gate. You may find your self depending on volumes and complexity of your dataflow(s) needing to even set this higher. The default 1024 is almost always guaranteed to be an issue.

/etc/security/limits.conf

*          hard    nproc    50000
*          soft    nproc    50000
*          hard    nofiles  50000
*          soft    nofiles  50000

/etc/security/limits.d/90-nproc.conf

*          hard    nproc    50000
*          soft    nproc    50000

Thanks,

Matt

View solution in original post

2 REPLIES 2

avatar
Super Mentor
@Joshua Adeleke

With NiFi dataflows it is typically to see large numbers of open file handles and user processes because of the concurrent thread operation supported by its many components.

In many cases you will find that the default ulimits for both open files and processes will fall short of what is needed for most dataflows.

I recommend setting these values to 50000 out the gate. You may find your self depending on volumes and complexity of your dataflow(s) needing to even set this higher. The default 1024 is almost always guaranteed to be an issue.

/etc/security/limits.conf

*          hard    nproc    50000
*          soft    nproc    50000
*          hard    nofiles  50000
*          soft    nofiles  50000

/etc/security/limits.d/90-nproc.conf

*          hard    nproc    50000
*          soft    nproc    50000

Thanks,

Matt

avatar
Expert Contributor

Thank you @Matt Clarke.