Member since
09-14-2018
32
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1006 | 10-18-2019 01:09 AM |
02-01-2022
08:19 PM
Spark uses the blacklist mechanism to enhance the scheduler’s ability to track failures. When a task fails on an executor, the blacklist module tracks the executor and host which has failed to execute the task. Beyond a threshold, the scheduler won't be able to schedule any more tasks on that node. If spark.blacklist.enabled is set to true, we need to always set the value of spark.blacklist.task.maxTaskAttemptsPerNode to greater than spark.task.maxFailures, else the Spark job will fail with the following error message:
ERROR util.Utils: Uncaught exception in thread main
java.lang.IllegalArgumentException: spark.blacklist.task.maxTaskAttemptsPerNode ( = 2) was >= spark.task.maxFailures ( = 2 ). Though blacklisting is enabled, with this configuration, Spark will not be robust to one bad node. Decrease spark.blacklist.task.maxTaskAttemptsPerNode, increase spark.task.maxFailures, or disable blacklisting with spark.blacklist.enabled.
If you try to trigger the following job from shell of edge node:
spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --conf spark.blacklist.enabled=true --conf spark.task.maxFailures=2 --conf spark.blacklist.task.maxTaskAttemptsPerNode=2 /opt/cloudera/parcels/CDH-{Version}/jars/spark-examples_{version}.jar 10
will result in the following error with job failed:
ERROR util.Utils: Uncaught exception in thread main java.lang.IllegalArgumentException: spark.blacklist.task.maxTaskAttemptsPerNode ( = 2) was >= spark.task.maxFailures ( = 2 ). Though blacklisting is enabled, with this configuration, Spark will not be robust to one bad node. Decrease spark.blacklist.task.maxTaskAttemptsPerNode, increase spark.task.maxFailures, or disable blacklisting with spark.blacklist.enabled
As clearly mentioned in the log message, the above error happens if you set spark.blacklist.task.maxTaskAttemptsPerNode >= spark.task.maxFailures.
You can resolve the above error by setting spark.task.maxFailures > spark.blacklist.task.maxTaskAttemptsPerNode as follows:
spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --conf spark.blacklist.enabled=true --conf spark.task.maxFailures=3 --conf spark.blacklist.task.maxTaskAttemptsPerNode=2 /opt/cloudera/parcels/CDH-{Version}/jars/spark-examples_{version}.jar 10
... View more
Labels:
10-30-2019
09:32 PM
Hi , We say the entire entry as region name. The region name is formed by the table name, the start key, the timestamp, and the encoded region name. table name :- img start key :- 0006943d-20150504220458043384375D00000002-00093 Timestamp :- 1527295748538 Encoded Region name :- 7b45a9f6f5584fc50b3152d41a5323a2 Please let me know if you have any doubts. Regards, Ganesh
... View more
10-18-2019
01:34 AM
Hi, Please export the KRB5_CONFIG path in shell and perform kinit as a valid user before running the command. export KRB5_CONFIG=/etc/krb5.conf kinit - as a valid user.. Run your command. Please try and share the results. Regards, Ganesh
... View more
10-18-2019
01:09 AM
Hi , I would suggest to do the following:- 1. Login to slave's /usr/bin. 2. Type the following command to check to which utility the spark-shell is linked. ls -al spark* It will print the following depending on your configuration. lrwxrwxrwx 1 root root 29 Oct 18 01:03 spark-shell -> /etc/alternatives/spark-shell lrwxrwxrwx 1 root root 30 Sep 30 03:11 spark-submit -> /etc/alternatives/spark-submit lrwxrwxrwx 1 root root 30 Sep 30 03:11 spark2-shell -> /etc/alternatives/spark2-shell lrwxrwxrwx 1 root root 31 Sep 30 03:11 spark2-submit -> /etc/alternatives/spark2-submit 3. Login to master shell /usr/bin: ln -s /etc/alternatives/spark-shell spark-shell The above command will create a link to the executable. Change the path "/etc/alternatives/spark-shell" as per your slave's configuration Please remember to login as root user. Let me know if it works. Thanks, Ganesh
... View more