question Previously working spark jobs only now throwing "java.io.IOException: Too many open files" error? in Support Questions

question Previously working spark jobs only now throwing "java.io.IOException: Too many open files" error? in Support Questions https://community.cloudera.com/t5/Support-Questions/Previously-working-spark-jobs-only-now-throwing-quot-java-io/m-p/295839#M217985 Running a HortonWork hadoop cluster (HDP-3.1.0.0) and getting a bunch of<BLOCKQUOTE>Failed on local exception: java.io.IOException: Too many open files</BLOCKQUOTE>errors when running spark jobs that up until this point have worked fine.I have seen many other questions like this where the answer is to increase the ulimit settings for open files and processes (this is also in <A href="https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/security-reference/content/kerberos_nonambari_limits_conf.html" target="_blank" rel="noopener">the HDP docs</A>) (and I'll note that I believe that mine are still at the system default settings), but... My question is: Why is this only happening now when previously the spark jobs have been running fine for months? The spark jobs I have been running have been running fine for months without incident and I have made no recent code changes. Don't know enough about the internals of spark to theorize about why things could be going wrong only now (would be odd to me if open files just build up in the course of running spark, but that seems like what is happening).Just as an example, just this code...<PRE>. . .sparkSession = SparkSession.builder.appName("GET_TABLE_COUNT").getOrCreate()sparkSession._jsc.sc().getExecutorMemoryStatus().keySet().size() . . .</PRE>now generates errors like...<PRE>. . . [2020-05-12 19:04:45,810] {bash_operator.py:128} INFO - 20/05/12 19:04:45 INFO Client: Application report for application_1579648183118_19918 (state: ACCEPTED) [2020-05-12 19:04:46,813] {bash_operator.py:128} INFO - 20/05/12 19:04:46 INFO Client: Application report for application_1579648183118_19918 (state: ACCEPTED) [2020-05-12 19:04:47,816] {bash_operator.py:128} INFO - 20/05/12 19:04:47 INFO Client: Application report for application_1579648183118_19918 (state: ACCEPTED) [2020-05-12 19:04:48,818] {bash_operator.py:128} INFO - 20/05/12 19:04:48 INFO Client: Application report for application_1579648183118_19918 (state: ACCEPTED) [2020-05-12 19:04:49,820] {bash_operator.py:128} INFO - 20/05/12 19:04:49 INFO Client: Application report for application_1579648183118_19918 (state: ACCEPTED) [2020-05-12 19:04:50,822] {bash_operator.py:128} INFO - 20/05/12 19:04:50 INFO Client: Application report for application_1579648183118_19918 (state: ACCEPTED) [2020-05-12 19:04:51,828] {bash_operator.py:128} INFO - 20/05/12 19:04:51 INFO Client: Application report for application_1579648183118_19918 (state: FAILED) [2020-05-12 19:04:51,829] {bash_operator.py:128} INFO - 20/05/12 19:04:51 INFO Client: [2020-05-12 19:04:51,829] {bash_operator.py:128} INFO - client token: N/A[2020-05-12 19:04:51,829] {bash_operator.py:128} INFO - diagnostics: Application application_1579648183118_19918 failed 2 times due to Error launching appattempt_1579648183118_19918_000002. Got exception: java.io.IOException: DestHost:destPort hw005.co.local:45454 , LocalHost:localPort hw001.co.local/172.18.4.46:0. Failed on local exception: java.io.IOException: Too many open files [2020-05-12 19:04:51,829] {bash_operator.py:128} INFO - at sun.reflect.GeneratedConstructorAccessor808.newInstance(Unknown Source) [2020-05-12 19:04:51,829] {bash_operator.py:128} INFO - at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)</PRE>My RAM and ulimit setting on the cluster look like...<PRE>[root@HW001]# clush -ab free -h---------------HW001--------------- total used free shared buff/cache availableMem: 31G 9.0G 1.1G 1.7G 21G 19G Swap: 8.5G 44K 8.5G ---------------HW002--------------- total used free shared buff/cache availableMem: 31G 7.3G 5.6G 568M 18G 22G Swap: 8.5G 308K 8.5G ---------------HW003--------------- total used free shared buff/cache availableMem: 31G 6.1G 4.0G 120M 21G 24G Swap: 8.5G 200K 8.5G ---------------HW004--------------- total used free shared buff/cache availableMem: 31G 2.9G 2.8G 120M 25G 27G Swap: 8.5G 28K 8.5G ---------------HW005--------------- total used free shared buff/cache availableMem: 31G 2.9G 4.6G 120M 23G 27G Swap: 8.5G 20K 8.5G ---------------airflowetl--------------- total used free shared buff/cache availableMem: 46G 5.3G 13G 2.4G 28G 38G Swap: 8.5G 124K 8.5G [root@HW001]# [root@HW001]# [root@HW001]# [root@HW001]# clush -ab ulimit -a ---------------HW[001-005] (5)--------------- core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127886 max locked memory (kbytes, -l) 64max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 127886 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ---------------airflowetl--------------- core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 192394 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 192394 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited</PRE>Don't know much about Hadoop admin, but just looking at the Ambari dashboard, the cluster does not seem to be overly taxed... <img src="https://community.cloudera.com/t5/image/serverpage/image-id/27468iB46537866306E3ED/image-size/medium?v=v2&px=400" role="button" title="Capture001.PNG" alt="Capture001.PNG" /> (though could not actually check the RM web UI, since it just throws a "too many open files" error).Anyone with more spark/hadoop experience know why this would be happening now? Wed, 13 May 2020 09:02:02 GMT rvillanueva 2020-05-13T09:02:02Z