Reply
Contributor
Posts: 26
Registered: ‎05-12-2015

ERROR YarnScheduler: Lost executor 2 on host08.rnd.company.net: remote Akka client

Hi Team,

 

Getting below error when opening SPARK-SHELL. Please help to fix.

 

 

15/07/07 12:56:58 ERROR YarnScheduler: Lost executor 2 on host08.rnd.company.net: remote Akka client disassociated
15/07/07 12:56:58 INFO DAGScheduler: Executor lost: 2 (epoch 0)
15/07/07 12:56:58 INFO BlockManagerMasterActor: Trying to remove executor 2 from BlockManagerMaster.
15/07/07 12:56:58 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(2, host08.rnd.company.net, 56270)
15/07/07 12:56:58 INFO BlockManagerMaster: Removed 2 successfully in removeExecutor
15/07/07 12:57:12 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@host06.rnd.company.net:40157/user/Executor#971639961] with ID 3
15/07/07 12:57:12 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@host08.rnd.company.net:51446/user/Executor#-172004619] with ID 4
15/07/07 12:57:12 INFO BlockManagerMasterActor: Registering block manager host06.rnd.company.net:48253 with 530.3 MB RAM, BlockManagerId(3, host06.rnd.company.net, 48253)
15/07/07 12:57:12 INFO BlockManagerMasterActor: Registering block manager host08.rnd.company.net:36027 with 530.3 MB RAM, BlockManagerId(4, host08.rnd.company.net, 36027)
15/07/07 12:57:23 ERROR YarnScheduler: Lost executor 3 on host06.rnd.company.net: remote Akka client disassociated
15/07/07 12:57:23 INFO DAGScheduler: Executor lost: 3 (epoch 0)
15/07/07 12:57:23 INFO BlockManagerMasterActor: Trying to remove executor 3 from BlockManagerMaster.
15/07/07 12:57:23 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(3, host06.rnd.company.net, 48253)
15/07/07 12:57:23 INFO BlockManagerMaster: Removed 3 successfully in removeExecutor
15/07/07 12:57:23 ERROR YarnScheduler: Lost executor 4 on host08.rnd.company.net: remote Akka client disassociated
15/07/07 12:57:23 INFO DAGScheduler: Executor lost: 4 (epoch 0)
15/07/07 12:57:23 INFO BlockManagerMasterActor: Trying to remove executor 4 from BlockManagerMaster.
15/07/07 12:57:23 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(4, host08.rnd.company.net, 36027)
15/07/07 12:57:23 INFO BlockManagerMaster: Removed 4 successfully in removeExecutor
15/07/07 12:57:38 INFO YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@blrrndnbudn02.rnd.company.net:56283/user/YarnAM#-1190889650]
15/07/07 12:57:38 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> host01.rnd.companay.net, PROXY_URI_BASES -> http://host01.rnd.acompany.net:8088/proxy/application_1436253584475_0002), /proxy/application_1436253584475_0002
15/07/07 12:57:38 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
15/07/07 12:57:41 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@host06.rnd.company.net:50001/user/Executor#-257950438] with ID 1
15/07/07 12:57:41 INFO DAGScheduler: Host added was in lost list earlier: host06.rnd.company.net
15/07/07 12:57:41 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@host08.rnd.company.net:33080/user/Executor#-971115393] with ID 2
15/07/07 12:57:41 INFO DAGScheduler: Host added was in lost list earlier: host08.rnd.company.net
15/07/07 12:57:41 INFO BlockManagerMasterActor: Registering block manager host06.rnd.company.net:59402 with 530.3 MB RAM, BlockManagerId(1, host06.rnd.company.net, 59402)
15/07/07 12:57:41 INFO BlockManagerMasterActor: Registering block manager host08.rnd.company.net:45339 with 530.3 MB RAM, BlockManagerId(2, host08.rnd.company.net, 45339)
15/07/07 12:57:52 ERROR YarnScheduler: Lost executor 1 on host06.rnd.company.net: remote Akka client disassociated
15/07/07 12:57:52 INFO DAGScheduler: Executor lost: 1 (epoch 0)
15/07/07 12:57:52 INFO BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster.
15/07/07 12:57:52 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(1, host06.rnd.company.net, 59402)
15/07/07 12:57:52 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor
15/07/07 12:57:52 ERROR YarnScheduler: Lost executor 2 on host08.rnd.company.net: remote Akka client disassociated
15/07/07 12:57:52 INFO DAGScheduler: Executor lost: 2 (epoch 0)
15/07/07 12:57:52 INFO BlockManagerMasterActor: Trying to remove executor 2 from BlockManagerMaster.
15/07/07 12:57:52 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(2, host08.rnd.company.net, 45339)
15/07/07 12:57:52 INFO BlockManagerMaster: Removed 2 successfully in removeExecutor
15/07/07 12:58:06 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@host06.rnd.company.net:50177/user/Executor#-353618613] with ID 3
15/07/07 12:58:06 INFO DAGScheduler: Host added was in lost list earlier: host06.rnd.company.net
15/07/07 12:58:06 INFO YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@host08.rnd.company.net:50401/user/Executor#171312342] with ID 4
15/07/07 12:58:06 INFO DAGScheduler: Host added was in lost list earlier: host08.rnd.company.net
15/07/07 12:58:06 INFO BlockManagerMasterActor: Registering block manager host06.rnd.company.net:43333 with 530.3 MB RAM, BlockManagerId(3, host06.rnd.company.net, 43333)
15/07/07 12:58:06 INFO BlockManagerMasterActor: Registering block manager host08.rnd.company.net:53676 with 530.3 MB RAM, BlockManagerId(4, host08.rnd.company.net, 53676)
15/07/07 12:58:17 ERROR YarnScheduler: Lost executor 3 on host06.rnd.company.net: remote Akka client disassociated
15/07/07 12:58:17 INFO DAGScheduler: Executor lost: 3 (epoch 0)
15/07/07 12:58:17 INFO BlockManagerMasterActor: Trying to remove executor 3 from BlockManagerMaster.
15/07/07 12:58:17 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(3, host06.rnd.company.net, 43333)
15/07/07 12:58:17 INFO BlockManagerMaster: Removed 3 successfully in removeExecutor
15/07/07 12:58:17 ERROR YarnScheduler: Lost executor 4 on host08.rnd.company.net: remote Akka client disassociated
15/07/07 12:58:17 INFO DAGScheduler: Executor lost: 4 (epoch 0)
15/07/07 12:58:17 INFO BlockManagerMasterActor: Trying to remove executor 4 from BlockManagerMaster.
15/07/07 12:58:17 INFO BlockManagerMasterActor: Removing block manager BlockManagerId(4, host08.rnd.company.net, 53676)
15/07/07 12:58:17 INFO BlockManagerMaster: Removed 4 successfully in removeExecutor
15/07/07 12:58:29 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
15/07/07 12:58:29 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
15/07/07 12:58:29 INFO SparkUI: Stopped Spark web UI at http://host01.rnd.company.net:4040
15/07/07 12:58:29 INFO DAGScheduler: Stopping DAGScheduler
15/07/07 12:58:29 INFO YarnClientSchedulerBackend: Shutting down all executors
15/07/07 12:58:29 INFO YarnClientSchedulerBackend: Asking each executor to shut down
15/07/07 12:58:29 INFO YarnClientSchedulerBackend: Stopped
15/07/07 12:58:29 INFO MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
15/07/07 12:58:29 INFO MemoryStore: MemoryStore cleared
15/07/07 12:58:29 INFO BlockManager: BlockManager stopped
15/07/07 12:58:29 INFO BlockManagerMaster: BlockManagerMaster stopped
15/07/07 12:58:29 INFO OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
15/07/07 12:58:29 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/07/07 12:58:29 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/07/07 12:58:29 INFO SparkContext: Successfully stopped SparkContext
15/07/07 12:58:29 INFO Remoting: Remoting shut down
15/07/07 12:58:29 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.

Cloudera Employee
Posts: 322
Registered: ‎01-16-2014

Re: ERROR YarnScheduler: Lost executor 2 on host08.rnd.company.net: remote Akka client

We have a separate forum for Spark related question. You will probably get more and quicker help there with Spark issues.

Since you are running on YARN you should check the yarn logs for the application. The files can be found via the Spark history server the host should provide links back to the yarn logs. Check what happened with the executor containers that should give you some further insight.

 

WIlfred