Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

spark application continuously running in Yarn

avatar
Rising Star

Hi Folks,

Hope all are doing well.

I'm newer in spark. I have installed HDP 2.6.2. i have added spark as a service. Before start the spark, There was no job running. But when i had started spark service, i have found two jobs are running continuously in UNDEFINED state.

#yarn application -list

18/05/28 15:07:51 INFO client.AHSProxy: Connecting to Application History server at 10.10.10.16:10200

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...

18/05/28 15:07:51 INFO client.RequestHedgingRMFailoverProxyProvider: Found active RM [rm2]

Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):4

Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1527494556086_0039 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.8:4040 application_1527494556086_0038 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive admin RUNNING UNDEFINED 10% http://10.10.10.12:4040

Thrift server is installed on both Server having IP are:

10.10.10.8

10.10.10.12

Can you please help me to clearify?

1 ACCEPTED SOLUTION

avatar
Super Guru

@Vinay K,

The 2 jobs which are running are Spark Thrift servers which will run as yarn applications. There is no need to worry. If you stop spark thrift servers then you won't see them running.

Spark2 thrift server will be running with app name "Thrift JDBC/ODBC Server"

Spark/Spark1 thrift server will be running with app name "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2".

.

Please "Accept" the answer if this helps.

.

-Aditya

View solution in original post

5 REPLIES 5

avatar
Super Guru

@Vinay K,

The 2 jobs which are running are Spark Thrift servers which will run as yarn applications. There is no need to worry. If you stop spark thrift servers then you won't see them running.

Spark2 thrift server will be running with app name "Thrift JDBC/ODBC Server"

Spark/Spark1 thrift server will be running with app name "org.apache.spark.sql.hive.thriftserver.HiveThriftServer2".

.

Please "Accept" the answer if this helps.

.

-Aditya

avatar
Rising Star

Thanks Aditya..

avatar
New Contributor

@Aditya Sirna


Hi Aditya, we are also interested in this answer. How do we find out what a job application like this is doing? Seems to run for days on my cluster.


Thanks much!

avatar
New Contributor

@Vinay

Hello Aditya, I'm also interested in this answer. How do we find out what a YARN application is doing running over many days?


thanks much!


avatar
Cloudera Employee

Hi, 

 

To understand what the Yarn application is doing, check the application logs of the particular yarn application and if the job hasnot completed , Also check for the Resource manager logs if it was stuck with any errors.

 

Thanks

Arun