Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master

Highlighted

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master

New Contributor

My pyspark applicatin used to working fine, but recently, I got the following error and i don't have any idea of the issue. Any help will be appreciated. Thank you.


Warning: Could not find the WD Fusion Client jars

Traceback (most recent call last):

File "/job_C/spark_sql.py", line 20, in <module>

spark = SparkSession.builder.appName("EXECUTING SQL QUERY").config(conf=conf).getOrCreate()

File "/app/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 173, in getOrCreate

File "/app/spark/python/lib/pyspark.zip/pyspark/context.py", line 334, in getOrCreate

File "/app/spark/python/lib/pyspark.zip/pyspark/context.py", line 118, in __init__

File "/app/spark/python/lib/pyspark.zip/pyspark/context.py", line 180, in _do_init

File "/app/spark/python/lib/pyspark.zip/pyspark/context.py", line 273, in _initialize_context

File "/app/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1525, in __call__

File "/app/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.

: org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)

at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)

at py4j.Gateway.invoke(Gateway.java:238)

at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)

at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)

at py4j.GatewayConnection.run(GatewayConnection.java:238)

at java.lang.Thread.run(Thread.java:748)