spark application continuously running in Yarn

[#1]

Hi Folks,

Hope all are doing well.

I'm newer in spark. I have installed HDP 2.6.2. i have added spark as a service. Before start the spark, There was no job running. But when i had started spark service, i have found two jobs are running continuously in UNDEFINED state.

#yarn application -list

18/05/28 15:07:51 INFO client.AHSProxy: Connecting to Application History server at 10.10.10.16:10200

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]

Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):4

Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1527494556086_0039 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.8:4040 application_1527494556086_0038 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.12:4040

Thrift server is installed on both Server having IP are:

10.10.10.8

10.10.10.12

Can you please help me to clearify?

[#2 · Accepted solution]

@Vinay K,

The 2 jobs which are running are Spark Thrift servers which will run as yarn applications. There is no need to worry. If you stop spark thrift servers then you won't see them running.

Spark2 thrift server will be running with app name "Thrift JDBC/ODBC Server"

Spark/Spark1 thrift server will be running with app name "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2".

.

Please "Accept" the answer if this helps.

.

-Aditya

[#3]

Thanks Aditya..

[#4]

@Aditya Sirna

Hi Aditya, we are also interested in this answer. How do we find out what a job application like this is doing? Seems to run for days on my cluster.

Thanks much!

[#5]

@Vinay

Hello Aditya, I'm also interested in this answer. How do we find out what a YARN application is doing running over many days?

thanks much!

[#6]

Hi, 

To understand what the Yarn application is doing, check the application logs of the particular yarn application and if the job hasnot completed , Also check for the Resource manager logs if it was stuck with any errors.

Thanks

Arun