Hello @Boron  I believe you are using HDP 3.x. Note that there is no Spark 1.x available in HDP 3. We need to use Spark 2.x. Set the SPARK_HOME to Spark 2. export SPARK_HOME=/usr/hdp/current/spark2-client ... View more

Hello @yagoaparecidoti , Can you please share the output of the below commands? From the Python shell: r0.headers["www-authenticate"] From the bash: # kinit # klist # date # curl -v -u: --negotiate -X GET http://<LIVY_NODE>:<PORT>/batches/ ... View more

We can also try running the below python code: 1. Run the kinit command. 2. Run the code in Python shell: import json, pprint, requests, textwrap from requests_kerberos import HTTPKerberosAuth host='http://localhost:8998' headers = {'Requested-By': 'livy','Content-Type': 'application/json','X-Requested-By': 'livy'} auth=HTTPKerberosAuth() data={'className': 'org.apache.spark.examples.SparkPi','jars': ["/tmp/spark-examples_2.11-2.4.7.7.1.7.1000-141.jar"],'name': 'livy-test1', 'file': 'hdfs:///tmp/spark-examples_2.11-2.4.7.7.1.7.1000-141.jar','args': ["10"]} r0 = requests.post(host + '/batches', data=json.dumps(data), headers=headers, auth=auth) r0.json()   ... View more

Hello @yagoaparecidoti : - Did we get the valid kerberos ticket before running the code? - Does the klist command show the valid expiry date? - Check the output of the below code after running your code: r0.headers["www-authenticate"]   Are we able to run the Sample Livy job using the CURL command?   Steps to run the sample job: 1. Copy the JAR to HDFS: # hdfs dfs -put /opt/cloudera/parcels/CDH/jars/spark-examples<VERSION>.jar /tmp 2. Make sure the JAR is present. # hdfs dfs -ls /tmp/ 3. CURL command to run the Spark job using Livy API. # curl -v -u: --negotiate -X POST --data '{"className": "org.apache.spark.examples.SparkPi", "jars": ["/tmp/spark-examples<VERSION>.jar"], "name": "livy-test", "file": "hdfs:///tmp/spark-examples<VERSION>.jar", "args": [10]}' -H "Content-Type: application/json" -H "X-Requested-By: User" http://<LIVY_NODE>:<PORT>/batches 4. Check for the running and completed Livy sessions. # curl http://<LIVY_NODE>:<PORT>/batches/ | python -m json.tool   NOTE:         * Change the JAR version ( <VERSION> ) according your CDP version.         * Replace the LIVY_NODE and PORT with the actual values.         * If you are running the cluster in secure mode, then make sure you have a valid Kerberos ticket and use the Kerberos authentication in curl command. ... View more

Hello @loridigia, I have tried to run the sample job like below and I see only one executor and one Driver container. # cd /usr/hdp/current/spark2-client # su spark $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 2 examples/jars/spark-examples*.jar 10000   Even the spark-shell is also limiting the containers to one when we are mentioning the --num-executors 1 $ spark-shell --num-executors 1   - What is the spark-submit command you are trying to run? - Are you seeing the same issue with the above sample job? ... View more

Hello @loridigia  I don't think there is a direct way to achieve this. But we have a workaround to do that. We can start the Spark jobs with Dynamic Allocation enabled. And we can set the Minimum executors to "0", initial executors to "1" and the idle timeout to "5s". With these configurations, the Spark job will start with 1 executor and after 5 seconds that container will be killed as it will be idle for more than 5 seconds. Now, we will have a Spark application only with the Driver / ApplicationMaster container running.   CONFIGS: --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.executorIdleTimeout=5s --conf spark.dynamicAllocation.initialExecutors=1 --conf spark.dynamicAllocation.maxExecutors=1 --conf spark.dynamicAllocation.minExecutors=1   NOTE: We can add these configs to the spark-defaults.conf so that the changes will be applied to all the Running jobs. Please be careful with other / actual Spark job configurations.   Make sure to mark the answer as the accepted solution. If it resolves your issue !  ... View more

- Do we have enabled SSL for the Spark UI? - If not, are we able to access the URL with the HTTP protocol? Also try with different browser and in Private Browsing mode. - Have we tried accessing the URL with the CURL command? - Do we have SPNEGO enabled for the cluster? ... View more

@Michael Bronson The warning message means that the Executor is unable to send the Heartbeat to the driver (might be network issue). This is just a warning message, but each failure increments heartbeat failure count and when we hit the maximum failures the executor will fail and exit with error. There are two configurations that we can tune to avoid this issue. spark.executor.heartbeat.maxFailures (default value: 60) Number of times an executor will try to send heartbeats to the driver before it gives up and exits (with exit code 56). spark.executor.heartbeatInterval ( default value: 10s ) Interval between each executor's heartbeats to the driver. Heartbeats let the driver know that the executor is still alive and update it with metrics for in-progress tasks. spark.executor.heartbeatInterval should be significantly less than spark.network.timeout ... View more