Running some queries e we get this error:   Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 102.0 failed 1 times, most recent failure: Lost task 11.0 in stage 102.0 (TID 4722, localhost, executor driver): java.io.FileNotFoundException: /tmp/blockmgr-95c88a5a-8d68-4b16-878d-158f40123999/1c/temp_shuffle_f82e49e7-a383-4803-82b3-030425703624 (Too many open files)   We modified limits.conf adding these value :   zeppelin soft nofile 32000 zeppelin hard nofile 32000   anyway looking at value of zeppelin process about open file limits the value is 4096   zeppelin@ ~]$ cat /proc/24222/limits Limit Soft Limit Hard Limit Units Max processes 510371 510371 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes [zeppelin@~]$   Is there some configuration file to set somewhere?   Thanks in advance   ... View more

On cloudera CDH 5.16.1 with spark2,  the Hive-on-Spark query containing WHERE conditions fail with the following error messages in the HiveServer2 log files: Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session <id> . .. Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session <id> .. .. Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error "Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream"   There is non problem with the same query from impala or   Hive-on-Spark  query without WHERE  conditions  ... View more