Member since
02-11-2016
2
Posts
0
Kudos Received
0
Solutions
10-14-2021
07:35 AM
Running some queries e we get this error:
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 102.0 failed 1 times, most recent failure: Lost task 11.0 in stage 102.0 (TID 4722, localhost, executor driver): java.io.FileNotFoundException: /tmp/blockmgr-95c88a5a-8d68-4b16-878d-158f40123999/1c/temp_shuffle_f82e49e7-a383-4803-82b3-030425703624 (Too many open files)
We modified limits.conf adding these value :
zeppelin soft nofile 32000 zeppelin hard nofile 32000
anyway looking at value of zeppelin process about open file limits the value is 4096
zeppelin@ ~]$ cat /proc/24222/limits Limit Soft Limit Hard Limit Units
Max processes 510371 510371 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes
[zeppelin@~]$
Is there some configuration file to set somewhere?
Thanks in advance
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache Zeppelin
01-22-2021
04:05 AM
On cloudera CDH 5.16.1 with spark2, the Hive-on-Spark query containing WHERE conditions fail with the following error messages in the HiveServer2 log files:
Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session <id> . .. Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session <id> .. .. Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error "Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream"
There is non problem with the same query from impala or Hive-on-Spark query without WHERE conditions
... View more
Labels: