question Re: spark application continuously running in Yarn in Archives of Support Questions (Read Only)

spark application continuously running in Yarn

vinayk — Mon, 28 May 2018 16:49:05 GMT

Hi Folks,

Hope all are doing well.

I'm newer in spark. I have installed HDP 2.6.2. i have added spark as a service. Before start the spark, There was no job running. But when i had started spark service, i have found two jobs are running continuously in UNDEFINED state.

#yarn application -list

18/05/28 15:07:51 INFO client.AHSProxy: Connecting to Application History server at 10.10.10.16:10200

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]

Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):4

Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1527494556086_0039 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.8:4040 application_1527494556086_0038 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.12:4040

Thrift server is installed on both Server having IP are:

10.10.10.8

10.10.10.12

Can you please help me to clearify?

Re: spark application continuously running in Yarn

asirna — Mon, 28 May 2018 17:43:31 GMT

@Vinay K,

The 2 jobs which are running are Spark Thrift servers which will run as yarn applications. There is no need to worry. If you stop spark thrift servers then you won't see them running.

Spark2 thrift server will be running with app name "Thrift JDBC/ODBC Server"

Spark/Spark1 thrift server will be running with app name "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2".

Please "Accept" the answer if this helps.

-Aditya

Re: spark application continuously running in Yarn

vinayk — Tue, 29 May 2018 15:22:40 GMT

Thanks Aditya..

Re: spark application continuously running in Yarn

Ricardo-DSA — Thu, 27 Jun 2019 01:43:43 GMT

@Aditya Sirna

Hi Aditya, we are also interested in this answer. How do we find out what a job application like this is doing? Seems to run for days on my cluster.

Thanks much!

Re: spark application continuously running in Yarn

Ricardo-DSA — Thu, 27 Jun 2019 07:53:12 GMT

@Vinay

Hello Aditya, I'm also interested in this answer. How do we find out what a YARN application is doing running over many days?

thanks much!

Re: spark application continuously running in Yarn

AKR — Wed, 30 Oct 2019 10:30:57 GMT

Hi,

To understand what the Yarn application is doing, check the application logs of the particular yarn application and if the job hasnot completed , Also check for the Resource manager logs if it was stuck with any errors.

Thanks

Arun