App timeline server failed to start due to webUI connection failed​

[#1]

Hello,

I have 5 node cluster using HDP2.4 and Ambari 2.2. Ambari server and App timeline server is on node1, Yarn Resource Manager is on node 3. App timeline server, Resource Manager and Node Manager fail because connection to their webUI port 8188, 8088, 8042 respectively failed. Node 1 has Ambari server on port 8080, which runs fine. But App timeline server 8188 on the same node can not connect.

No job is running. Just installed all the components and found alerts of those connection failures on Ambari web interface cluster monitoring.

OS is centos 7, firewalld and selinux are off.

Anyone knows why or how to fix this?

Thanks in advance!

Shannon

[#2]

@Emil

Hi Emil,

Thanks for replying. Sorry I was out of office. Found error in yarn-yarn-timelineserver-xxx.log only (see below). It seems to be missing a timeline plugin related to tez. Ambari used is the newest as of March 2016 and should not have any patch issue related to this, should it? Tez was not running as hive was not used. Do you know how this can be fixed?

Thanks!

Shannon

--------------copied from log---------

2016-03-10 20:57:02,239 INFO timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:loadPlugIns(217)) - Error loading plugin org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl org.apache.hadoop.yarn.exceptions.YarnException: Unable to load class: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:187) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:214) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:183) ... 8 more

2016-03-10 20:57:02,245 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(171)) - Error starting ApplicationHistoryServer java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:223) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)

[#3]

@Emil

Hi Emil,

Turn on Hive with Tez as execution engine. Still same problem. Where should I get this missing plugin, do you know?

Thanks!

Shannon

[#4]

@shannon luo

I am a solution engineer located in King of Prussia and trying to reproduce the problem you have. Could you please provide some more information about the environment of you cluster

• public cloud/private cloud/vm
• kernal version of centos7
• Java version
• node layout for where all the services you have
  • Ambari server
  • name nodes
  • resource manager nodes
  • hive/tez/hbase/spark/..
  • clients

I assume you following the instruction on the following page during you installation and have prepared the environments following the instruction.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Installing_HDP_AMB/content/ch_Getting_Rea...

Thanks,

Qi

[#5]

@Qi Wang

Hi Qi,

would you please email me at sluo(at)anexinet.com? I can send environment details to your email.

Thanks!

Shannon

[#6 · Accepted solution]

Thanks to help from @Qi Wang, the problem is solved by moving the appTimeline server to the same server where history server is. Thank you, Qi!

[#7]

Hi, I am new to HDP and I am encountering the same error.

May I know where do I see the history server is?

Could you list down the steps you used to "move the appTimeline server to the same server where the history server is"?

Thank you!

[#8]

@qi

@Qi Wang I had a similar problem with the yarn resource manager running on master 2 not being able to connect to yarn App timeline server running on master 3. I moved yarnApp timeline server to master 2 and it works fine now. But just wanted to check does Apptimeline server have to be on the same node as as yarn rm or How are the three components related App timeline server, YARN RM and History server ?

Error logs

resource_management.core.exceptions.Fail: Execution of 'yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000' returned 2. 16/05/02 15:44:15 INFO impl.TimelineClientImpl: Timeline service address: http://master3.com:8188/ws/v1/timeline/