Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.
CLDR
Cloudera Employee
My Accepted Solutions
Show All

Configuring spark.task.maxFailures & spark.blackli...

by Cloudera Employee in Community Articles
‎02-01-2022 08:19 PM
‎02-01-2022 08:19 PM
Spark uses the blacklist mechanism to enhance the scheduler’s ability to track failures. When a task fails on an executor, the blacklist module tracks the executor and host which has failed to execute the task. Beyond a threshold, the scheduler won't be able to schedule any more tasks on that node. If spark.blacklist.enabled is set to true, we need to always set the value of spark.blacklist.task.maxTaskAttemptsPerNode to greater than spark.task.maxFailures, else the Spark job will fail with the following error message: ERROR util.Utils: Uncaught exception in thread main java.lang.IllegalArgumentException: spark.blacklist.task.maxTaskAttemptsPerNode ( = 2) was >= spark.task.maxFailures ( = 2 ). Though blacklisting is enabled, with this configuration, Spark will not be robust to one bad node. Decrease spark.blacklist.task.maxTaskAttemptsPerNode, increase spark.task.maxFailures, or disable blacklisting with spark.blacklist.enabled. If you try to trigger the following job from shell of edge node: spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --conf spark.blacklist.enabled=true --conf spark.task.maxFailures=2 --conf spark.blacklist.task.maxTaskAttemptsPerNode=2 /opt/cloudera/parcels/CDH-{Version}/jars/spark-examples_{version}.jar 10 will result in the following error with job failed: ERROR util.Utils: Uncaught exception in thread main java.lang.IllegalArgumentException: spark.blacklist.task.maxTaskAttemptsPerNode ( = 2) was >= spark.task.maxFailures ( = 2 ). Though blacklisting is enabled, with this configuration, Spark will not be robust to one bad node. Decrease spark.blacklist.task.maxTaskAttemptsPerNode, increase spark.task.maxFailures, or disable blacklisting with spark.blacklist.enabled As clearly mentioned in the log message, the above error happens if you set   spark.blacklist.task.maxTaskAttemptsPerNode >= spark.task.maxFailures.   You can resolve the above error by setting spark.task.maxFailures > spark.blacklist.task.maxTaskAttemptsPerNode as follows: spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --conf spark.blacklist.enabled=true --conf spark.task.maxFailures=3 --conf spark.blacklist.task.maxTaskAttemptsPerNode=2 /opt/cloudera/parcels/CDH-{Version}/jars/spark-examples_{version}.jar 10 ... View more
Labels:

Re: Hbase Region Name

by Cloudera Employee in Support Questions
‎10-30-2019 09:32 PM
‎10-30-2019 09:32 PM
Hi ,          We say the entire entry as region name.        The region name is formed by the table name, the start key, the timestamp, and the encoded region name.          table name :- img        start key     :-   0006943d-20150504220458043384375D00000002-00093        Timestamp :-  1527295748538        Encoded Region name :-  7b45a9f6f5584fc50b3152d41a5323a2       Please let me know if you have any doubts.   Regards, Ganesh ... View more

Re: Remote beeline client from Mac to a secure CDH...

by Cloudera Employee in Support Questions
‎10-18-2019 01:34 AM
‎10-18-2019 01:34 AM
Hi,           Please export the KRB5_CONFIG path in shell and perform kinit as a valid user before running the command.            export KRB5_CONFIG=/etc/krb5.conf kinit  - as a valid user..   Run your command. Please try and share the results.   Regards, Ganesh ... View more

Re: how to recover after master /user/bin was dele...

by Cloudera Employee in Support Questions
‎10-18-2019 01:09 AM
‎10-18-2019 01:09 AM
Hi ,         I would suggest to do the following:-              1. Login to slave's /usr/bin.            2. Type the following command to check to which utility the spark-shell is linked.                        ls -al spark*                   It will print the following  depending on your configuration.                       lrwxrwxrwx 1 root root 29 Oct 18 01:03 spark-shell -> /etc/alternatives/spark-shell lrwxrwxrwx 1 root root 30 Sep 30 03:11 spark-submit -> /etc/alternatives/spark-submit lrwxrwxrwx 1 root root 30 Sep 30 03:11 spark2-shell -> /etc/alternatives/spark2-shell lrwxrwxrwx 1 root root 31 Sep 30 03:11 spark2-submit -> /etc/alternatives/spark2-submit           3. Login to master shell /usr/bin:                      ln -s   /etc/alternatives/spark-shell spark-shell               The above command will create a link to the executable. Change the path "/etc/alternatives/spark-shell" as per your slave's configuration                 Please remember to login as root user.               Let me know if it works.   Thanks, Ganesh ... View more
Contact Me
Online
Offline
Last Visited
‎12-06-2022 06:40 PM
Public Statistics
Date Registered ‎09-14-2018 03:48 AM
Last Visited ‎12-06-2022 06:40 PM
Total Messages Posted 32
Total Kudos Received 2