Support Questions
Find answers, ask questions, and share your expertise

spark application continuously running in Yarn

Solved Go to solution

spark application continuously running in Yarn

Expert Contributor

Hi Folks,

Hope all are doing well.

I'm newer in spark. I have installed HDP 2.6.2. i have added spark as a service. Before start the spark, There was no job running. But when i had started spark service, i have found two jobs are running continuously in UNDEFINED state.

#yarn application -list

18/05/28 15:07:51 INFO client.AHSProxy: Connecting to Application History server at 10.10.10.16:10200

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]

Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):4

Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1527494556086_0039 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.8:4040 application_1527494556086_0038 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.12:4040

Thrift server is installed on both Server having IP are:

10.10.10.8

10.10.10.12

Can you please help me to clearify?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: spark application continuously running in Yarn

@Vinay K,

The 2 jobs which are running are Spark Thrift servers which will run as yarn applications. There is no need to worry. If you stop spark thrift servers then you won't see them running.

Spark2 thrift server will be running with app name "Thrift JDBC/ODBC Server"

Spark/Spark1 thrift server will be running with app name "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2".

.

Please "Accept" the answer if this helps.

.

-Aditya

View solution in original post

5 REPLIES 5

Re: spark application continuously running in Yarn

@Vinay K,

The 2 jobs which are running are Spark Thrift servers which will run as yarn applications. There is no need to worry. If you stop spark thrift servers then you won't see them running.

Spark2 thrift server will be running with app name "Thrift JDBC/ODBC Server"

Spark/Spark1 thrift server will be running with app name "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2".

.

Please "Accept" the answer if this helps.

.

-Aditya

View solution in original post

Re: spark application continuously running in Yarn

Expert Contributor

Thanks Aditya..

Re: spark application continuously running in Yarn

New Contributor

@Aditya Sirna


Hi Aditya, we are also interested in this answer. How do we find out what a job application like this is doing? Seems to run for days on my cluster.


Thanks much!

Re: spark application continuously running in Yarn

New Contributor

@Vinay

Hello Aditya, I'm also interested in this answer. How do we find out what a YARN application is doing running over many days?


thanks much!


Re: spark application continuously running in Yarn

Cloudera Employee

Hi, 

 

To understand what the Yarn application is doing, check the application logs of the particular yarn application and if the job hasnot completed , Also check for the Resource manager logs if it was stuck with any errors.

 

Thanks

Arun