java.io.FileNotFoundException: Too many open files error

[#1]

Hi All,

We have been facing issues while Abinitio jobs being run. Here is some outline about the setup. In our end, Hadoop is used only for storage, remaining part is taken care by Abinitio (they ve product for data ingestion and retrieval). When they try to ingest the data (all write operation) & ran some query to read some tables we have been hit with the below error.

I have checked and changed the ulimit ( -n ) value for abiuser which they have been using to run the job for data ingestion / read from 1024 to 20,000 on all data nodes and name node. Also verified hdfs limits on all data nodes and it is like below. HDP version is 2.4. There are no errors on the Ambari console as well.

hdfs - nofile 128000 hdfs - nproc 65536

[#2 · Accepted solution]

@Muthukumar S

Assuming that you run a secure cluster, if the DataNodes are started as the root user, JSVC downgrades the processing using setuid to hdfs. However, the ulimit is based on the ulimit of the root user, and the default ulimit values assigned to the root user for the maximum number of open files and processes may be too low for a secure cluster.

From what you provided above, you increased the limits for abiuser user.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729...

If this is helpful, please vote this response.