Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Getting lot of ERROR ErrorMonitor: AssociationError in spark 1.5.0 application ( cloudera 5.5.0)

Getting lot of ERROR ErrorMonitor: AssociationError in spark 1.5.0 application ( cloudera 5.5.0)

Hi

 

I have installed clouera 5.5.0 with spark 1.5.0 .

when i run my application in yarn-client mode i am getting lot of ErrorMonitor: AssociationError Error. 

 

ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.44.101:43122] <- [akka.tcp://driverPropsFetcher@ivcp-m06.novalocal:41531]: Error [Shut down address: akka.tcp://driverPropsFetcher@ivcp-m06.novalocal:41531] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@ivcp-m06.novalocal:41531
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
15/11/30 17:17:02 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.44.101:43122] <- [akka.tcp://driverPropsFetcher@ivcp-m01.novalocal:37308]: Error [Shut down address: akka.tcp://driverPropsFetcher@ivcp-m01.novalocal:37308] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@ivcp-m01.novalocal:37308
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$

 

ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.44.101:43122] <- [akka.tcp://sparkExecutor@ivcp-m06.novalocal:46082]: Error [Shut down address: akka.tcp://sparkExecutor@ivcp-m06.novalocal:46082] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@ivcp-m06.novalocal:46082
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.
]
akka.event.Logging$Error$NoCause$
15/11/30 17:17:06 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.44.101:43122] <- [akka.tcp://sparkExecutor@ivcp-m01.novalocal:43972]: Error [Shut down address: akka.tcp://sparkExecutor@ivcp-m01.novalocal:43972] [
akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@ivcp-m01.novalocal:43972
Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down.

 

 

can you please help me to resolve these error.

 

Regards

Prateek
]

3 REPLIES 3

Re: Getting lot of ERROR ErrorMonitor: AssociationError in spark 1.5.0 application ( cloudera 5.5.0)

Master Collaborator

Does this happen during shutdown? I have seen it during shutdown and it's noisy but ignorable.

Re: Getting lot of ERROR ErrorMonitor: AssociationError in spark 1.5.0 application ( cloudera 5.5.0)

Hi

 

yes its happen during shutdown ..

but i have not seen this error in cloudera 5.4.7 version . i have tested same application in both enviroment.

 

Regards

Prateek

Re: Getting lot of ERROR ErrorMonitor: AssociationError in spark 1.5.0 application ( cloudera 5.5.0)

New Contributor

Duplicate issue at:  https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/spark-error-after-upgrade-to-cdh5-...

 

Notice my comments that this is not exculsively at shutdown.  We are consistently seeing it at startup but usually Spark retries (defaults to 3 retries) and associates with Executor actors.  This makes me think there is a timing issue but the error messages suggest the Executor actors are actually shuttingdown.  

 

Looking at the logs for the executors I see no reason for the shutdown.  Perhaps I need to set a particular log level to DEBUG?  I have all os org.apache.spark set to DEBUG right now.  Maybe it's an Akka log level? 

Don't have an account?
Coming from Hortonworks? Activate your account here