Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

App timeline server failed to start due to webUI connection failed​

avatar
Contributor

Hello,

I have 5 node cluster using HDP2.4 and Ambari 2.2. Ambari server and App timeline server is on node1, Yarn Resource Manager is on node 3. App timeline server, Resource Manager and Node Manager fail because connection to their webUI port 8188, 8088, 8042 respectively failed. Node 1 has Ambari server on port 8080, which runs fine. But App timeline server 8188 on the same node can not connect.

No job is running. Just installed all the components and found alerts of those connection failures on Ambari web interface cluster monitoring.

OS is centos 7, firewalld and selinux are off.

Anyone knows why or how to fix this?

Thanks in advance!

Shannon

1 ACCEPTED SOLUTION

avatar
Contributor

Thanks to help from @Qi Wang, the problem is solved by moving the appTimeline server to the same server where history server is. Thank you, Qi!

View solution in original post

7 REPLIES 7

avatar
Contributor

@Emil

Hi Emil,

Thanks for replying. Sorry I was out of office. Found error in yarn-yarn-timelineserver-xxx.log only (see below). It seems to be missing a timeline plugin related to tez. Ambari used is the newest as of March 2016 and should not have any patch issue related to this, should it? Tez was not running as hive was not used. Do you know how this can be fixed?

Thanks!

Shannon

--------------copied from log---------

2016-03-10 20:57:02,239 INFO timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:loadPlugIns(217)) - Error loading plugin org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl org.apache.hadoop.yarn.exceptions.YarnException: Unable to load class: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:187) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:214) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:183) ... 8 more

2016-03-10 20:57:02,245 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(171)) - Error starting ApplicationHistoryServer java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:223) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)

avatar
Contributor

@Emil

Hi Emil,

Turn on Hive with Tez as execution engine. Still same problem. Where should I get this missing plugin, do you know?

Thanks!

Shannon

avatar
Master Collaborator

@shannon luo

I am a solution engineer located in King of Prussia and trying to reproduce the problem you have. Could you please provide some more information about the environment of you cluster

  • public cloud/private cloud/vm
  • kernal version of centos7
  • Java version
  • node layout for where all the services you have
    • Ambari server
    • name nodes
    • resource manager nodes
    • hive/tez/hbase/spark/..
    • clients

I assume you following the instruction on the following page during you installation and have prepared the environments following the instruction.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Installing_HDP_AMB/content/ch_Getting_Rea...

Thanks,

Qi

avatar
Contributor

@Qi Wang

Hi Qi,

would you please email me at sluo(at)anexinet.com? I can send environment details to your email.

Thanks!

Shannon

avatar
Contributor

Thanks to help from @Qi Wang, the problem is solved by moving the appTimeline server to the same server where history server is. Thank you, Qi!

avatar
New Contributor

Hi, I am new to HDP and I am encountering the same error.

May I know where do I see the history server is?

Could you list down the steps you used to "move the appTimeline server to the same server where the history server is"?

Thank you!

avatar
New Contributor

@qi

@Qi Wang I had a similar problem with the yarn resource manager running on master 2 not being able to connect to yarn App timeline server running on master 3. I moved yarnApp timeline server to master 2 and it works fine now. But just wanted to check does Apptimeline server have to be on the same node as as yarn rm or How are the three components related App timeline server, YARN RM and History server ?

Error logs

resource_management.core.exceptions.Fail: Execution of 'yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000' returned 2. 16/05/02 15:44:15 INFO impl.TimelineClientImpl: Timeline service address: http://master3.com:8188/ws/v1/timeline/