Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Who agreed with this topic

Long running yarn app (storm-yarn) exits with Invalid AMRMToken after some random number of hours.

Explorer

Hi I'm running storm-yarn on CDH5.3.2 and it's working great, except that it exits seemingly at random after running for between a few hours and a few days.

 

I see an exception in the yarn log, followed by a graceful shutdown:

 

15/04/16 10:58:15 ERROR yarn.MasterServer: Unhandled error in AM:
org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1429008207550_0003_000002
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
              at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
              at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
              at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
              at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
              at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
              at com.sun.proxy.$Proxy12.allocate(Unknown Source)
              at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
              at com.yahoo.storm.yarn.MasterServer$1.run(MasterServer.java:69)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1429008207550_0003_000002
              at org.apache.hadoop.ipc.Client.call(Client.java:1411)
              at org.apache.hadoop.ipc.Client.call(Client.java:1364)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
              at com.sun.proxy.$Proxy11.allocate(Unknown Source)
              at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
              ... 8 more
15/04/16 10:58:15 INFO yarn.StormMasterServerHandler: stopping supervisors...
15/04/16 10:58:15 INFO yarn.StormMasterServerHandler: stopping UI...
15/04/16 10:58:15 INFO yarn.StormMasterServerHandler: stopping nimbus...

 

This will be a blocking issue for deploying Cloudera. Other options include Hortonworks who claim to support storm-yarn but we have significant CDH knowledge and don't want to switch, or stand-alone storm which we do have experience with but would rather run a single cluster.

 

Any help would be awesome,

 

Thanks,

James

Who agreed with this topic