Support Questions
Find answers, ask questions, and share your expertise

Zeppelin max open file limits

New Contributor

Running some queries e we get this error:


Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 102.0 failed 1 times, most recent failure: Lost task 11.0 in stage 102.0 (TID 4722, localhost, executor driver): /tmp/blockmgr-95c88a5a-8d68-4b16-878d-158f40123999/1c/temp_shuffle_f82e49e7-a383-4803-82b3-030425703624 (Too many open files)


We modified limits.conf adding these value :


zeppelin soft nofile 32000
zeppelin hard nofile 32000


anyway looking at value of zeppelin process about open file limits the value is 4096


zeppelin@ ~]$ cat /proc/24222/limits
Limit Soft Limit Hard Limit Units

Max processes 510371 510371 processes
Max open files 4096 4096 files
Max locked memory 65536 65536 bytes



Is there some configuration file to set somewhere?


Thanks in advance



Expert Contributor

Hi @Paop 


We don't have enough information (how much data, spark submit command etc) to provide solution. Please raise a case for this issue. 

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.