https://community.cloudera.com/t5/Support-Questions/Previously-working-spark-jobs-only-now-throwing-quot-java-io/m-p/295850#M217993 <P>your PB may be caused by caused by several reasons</P><P>&nbsp;</P><P>firstly i think 1024 is not enough, you should increase it</P><P>&nbsp;</P><P>opened files may be increasing day after day ( an application may stream more data from/into splitted files)</P><P>a spark application may also import/open more libraries today</P><P>etc...</P><P>&nbsp;</P><P>please check opened file by the user (that runs spark jobs) to find the possible cause</P><P><EM>lsof -u myUser&nbsp;</EM>( | wc -l ... )</P><P>&nbsp;</P><P>check&nbsp;<EM>lsof (lsof +D directory) , and find how many opened files per job and how many jobs are runing etc...</EM></P><P>&nbsp;</P><P>&nbsp;</P> Wed, 13 May 2020 11:56:55 GMT rachid-berkane 2020-05-13T11:56:55Z