question No Completed Application Found in Spark History Server in Archives of Support Questions (Read Only)

No Completed Application Found in Spark History Server

Pal — Fri, 16 Sep 2022 09:13:37 GMT

I have installed CDH 5.2.0 on a single node

When launch spark python and after successfully executing simple python sc.parallelize(range(1000)).count()

I don’t see the this application in spark history server http://localhost:18088, when open I can only see

Event Log Location : hdfs//localhost:8020/user/spark/applicationHistory

No Completed Application Found

Can you please advise me if I have check any configurations or

I appreciate any help on this

thanks

Pal

Re: No Completed Application Found in Spark History Server

srowen — Wed, 19 Nov 2014 08:13:06 GMT

Are you running Spark on YARN, or using Spark standalone? if the latter, you won't see any YARN history since it's not using YARN.

Re: No Completed Application Found in Spark History Server

Pal — Fri, 21 Nov 2014 00:55:27 GMT

Hi Sowen,

Thanks for your reply,

Are you running Spark on YARN, or using Spark standalone? if the latter, you won't see any YARN history since it's not using YARN.

Yes Spark is running on YARN (MR2 Included) I checked this in Cloudera Manager Web console Spark à Configuration

Is that mean I have to configure Spark to user YARN (MR2 Included) according in Cloudera Manager its already OR am I missing something

I did a default installation and followed wizard.

Can you please advice

thanks

Pal

Re: No Completed Application Found in Spark History Server

srowen — Fri, 21 Nov 2014 09:07:17 GMT

Yes but did you also submit your Spark app to YARN? what is your master for the app?

Re: No Completed Application Found in Spark History Server

Pal — Mon, 24 Nov 2014 09:58:38 GMT

Hi Sowen,

Below are the steps

I launched “pyspark” command shell by executing “/opt/cloudera/parcels/CDH/bin/pyspark”
In pyspark command I executed simple python “sc.parallelize(range(1000)).count()” , this simple one line program successfully in pyspark command shell. Since Apache Spark in CDH5.2 is configured to run on YARN , I was expecting to see the pyspark app in spark history server http://localhost:18088

Since its single node installation master and worker are in same node

Please advice if I am running this simple python program incorrectly

thanks

Pal

Re: No Completed Application Found in Spark History Server

srowen — Mon, 24 Nov 2014 10:01:23 GMT

Spark defaults to run with a local master IIRC. You should set "--master yarn-client" to actually use YARN. I assume it's not different for pyspark vs spark-shell.

Re: No Completed Application Found in Spark History Server

Pal — Mon, 24 Nov 2014 22:22:00 GMT

Hi Srowen,

Thanks for the update, I am new to spark, can you please guide where should I set "--master yarn-client" in configuration file or ?

thanks

Pal

Re: No Completed Application Found in Spark History Server

srowen — Mon, 24 Nov 2014 22:37:22 GMT

This is an option to spark-submit or pyspark. Look at the Spark docs.

Re: No Completed Application Found in Spark History Server

AKR — Fri, 26 Jul 2019 16:59:34 GMT

Hi Pal,

Can you grep for the particular application ID in the folder /user/spark/applicationHistory to make sure whether the job has been successfully completed or still in .inprogress state?

Thanks

AKR