question RUNNING applications and Incomplete applications seems hang in Support Questions

RUNNING applications and Incomplete applications seems hang

diablo2 — Sat, 24 Sep 2016 12:51:33 GMT

As a newbie of hadoop, I encounter below trouble. I found several RUNNING applications in Yarn resource manager UI those hang and never complete.

Show 20406080100 entries Search:

ID	User	Name	Application Type	Queue	StartTime	FinishTime	State	FinalStatus	Running Containers
application_1474533507895_0024	hive	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	SPARK	default	Fri Sep 23 11:37:55 +0800 2016	N/A	RUNNING	UNDEFINED	ApplicationMaster
application_1474514425259_0011	hive	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	SPARK	default	Thu Sep 22 11:25:54 +0800 2016	N/A	RUNNING	UNDEFINED	ApplicationMaster
application_1474514425259_0010	hive	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	SPARK	default	Thu Sep 22 11:25:54 +0800 2016	N/A	RUNNING	UNDEFINED	ApplicationMaster
application_1474514425259_0009	hive	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	SPARK	default	Thu Sep 22 11:25:54 +0800 2016	N/A	RUNNING	UNDEFINED	ApplicationMaster
application_1474514425259_0006	hive	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	SPARK	default	Thu Sep 22 11:25:53 +0800 2016	N/A	RUNNING	UNDEFINED	ApplicationMaster

Showing 1 to 5 of 5 entries FirstPrevious1Next

Last

and in spark history server UI

1.6.2 History Server

Event log directory: hdfs:///spark-history

Showing 1-9 of 9 (Incomplete applications)1

App ID	App Name	Started	Completed	Duration	Spark User
application_1474533507895_0024	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/23 11:37:52	-	-	hive
application_1474514425259_0011	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/22 11:25:51	-	-	hive
application_1474514425259_0010	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/22 11:25:51	-	-	hive
application_1474514425259_0009	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/22 11:25:50	-	-	hive
application_1474514425259_0006	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/22 11:25:49	-	-	hive
application_1474466529970_0009	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/21 22:15:33	-	-	hive
application_1474466529970_0008	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/21 22:15:28	-	-	hive
application_1474466529970_0006	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/21 22:15:28	-	-	hive
application_1474466529970_0007	org.apache.spark.sql.hive.thriftserver.HiveThriftServer2	2016/09/21 22:15:27	-	-	hive

Back to completed applications

How can I do with those above running applications? how to know the root cause of such issue ? and how to kill them and delete them?

Tons of thanks.

Re: RUNNING applications and Incomplete applications seems hang

tmccuch — Sat, 24 Sep 2016 21:42:45 GMT

@Huahua Wei You need to explicitly stop the SparkContext sc by calling sc.stop. In cluster settings if you don't explicitly call sc.stop() your application may hang. Like closing files, network connections, etc, when you're done with them, it's a good idea to call sc.stop(), which lets the spark master know that your application is finished consuming resources. If you don't call sc.stop(), the event log information that is used by the history server will be incomplete, and your application will not show up in the history server's UI.

Re: RUNNING applications and Incomplete applications seems hang

TimothySpann — Sat, 24 Sep 2016 22:36:00 GMT

you can kill them in YARN as well if they are hung, but follow Tom's advice first. stop and clean up your jobs.

Re: RUNNING applications and Incomplete applications seems hang

diablo2 — Mon, 26 Sep 2016 14:25:51 GMT

I called sc.stop() in spark-shell, but seem no help. the running applications still there.

scala> sc.stop() 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 16/09/26 15:13:17 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 16/09/26 15:13:17 INFO SparkUI: Stopped Spark web UI at http://202.1.2.138:4041 16/09/26 15:13:17 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/09/26 15:13:17 INFO MemoryStore: MemoryStore cleared 16/09/26 15:13:17 INFO BlockManager: BlockManager stopped 16/09/26 15:13:17 INFO BlockManagerMaster: BlockManagerMaster stopped 16/09/26 15:13:17 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/09/26 15:13:17 INFO SparkContext: Successfully stopped SparkContext

scala> 16/09/26 15:13:17 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 16/09/26 15:13:17 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/09/26 15:13:17 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down. sc.stop() 16/09/26 15:15:57 INFO SparkContext: SparkContext already stopped.

scala> sc.stop() 16/09/26 15:18:36 INFO SparkContext: SparkContext already stopped.

scala>

Yarn KILL, can work.

[root@insightcluster133 /]# yarn application -list 16/09/26 15:21:08 INFO impl.TimelineClientImpl: Timeline service address: http://insightcluster132.huawei.com:8188/ws/v1/timeline/ 16/09/26 15:21:08 INFO client.RMProxy: Connecting to ResourceManager at insightcluster133.huawei.com/202.1.2.133:8050 16/09/26 15:21:08 INFO client.AHSProxy: Connecting to Application History server at insightcluster132.huawei.com/202.1.2.132:10200 Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):3 Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL application_1474533507895_0024 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive default RUNNING UNDEFINED 10% http://202.1.2.138:4040 application_1474514425259_0010 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive default RUNNING UNDEFINED 10% http://202.1.2.130:4040 application_1474514425259_0009 org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 SPARK hive default RUNNING UNDEFINED 10% http://202.1.2.134:4040 [root@insightcluster133 /]# yarn application -kill application_1474514425259_0009 16/09/26 15:21:21 INFO impl.TimelineClientImpl: Timeline service address: http://insightcluster132.huawei.com:8188/ws/v1/timeline/ 16/09/26 15:21:21 INFO client.RMProxy: Connecting to ResourceManager at insightcluster133.huawei.com/202.1.2.133:8050 16/09/26 15:21:22 INFO client.AHSProxy: Connecting to Application History server at insightcluster132.huawei.com/202.1.2.132:10200 Killing application application_1474514425259_0009 16/09/26 15:21:22 INFO impl.YarnClientImpl: Killed application application_1474514425259_0009 [root@insightcluster133 /]#

But, the killed items still in the History Server UI

Re: RUNNING applications and Incomplete applications seems hang

diablo2 — Tue, 27 Sep 2016 13:40:09 GMT

I called sc.stop() in spark-shell, but seem no help. the running applications still there.

scala> sc.stop() 16/09/26 15:18:36 INFO SparkContext: SparkContext already stopped.

scala>

Yarn KILL, can work.

But, the killed items still in the History Server UI

Re: RUNNING applications and Incomplete applications seems hang

tmccuch — Sat, 01 Oct 2016 07:01:10 GMT

@Huahua Wei What version of Spark are you running? There is a JIRA for Spark 1.5.1 where the SparkContext stop method does not close HiveContexts.

Re: RUNNING applications and Incomplete applications seems hang

diablo2 — Sun, 09 Oct 2016 14:45:35 GMT

Spark 1.6.x.2.5

Re: RUNNING applications and Incomplete applications seems hang

AKR — Tue, 03 Sep 2019 18:19:50 GMT

Hi,

We need to review the Resource manager logs to look for the Errors if any,. Also we need to view the Resource manager webUI to check for the resource utilization and Memory utiilization in queue wise on the jobs submitted.

Thanks

AKR