question Re: App timeline server failed to start due to webUI connection failed in Support Questions

App timeline server failed to start due to webUI connection failed

Luos — Wed, 23 Mar 2016 04:12:50 GMT

Hello,

I have 5 node cluster using HDP2.4 and Ambari 2.2. Ambari server and App timeline server is on node1, Yarn Resource Manager is on node 3. App timeline server, Resource Manager and Node Manager fail because connection to their webUI port 8188, 8088, 8042 respectively failed. Node 1 has Ambari server on port 8080, which runs fine. But App timeline server 8188 on the same node can not connect.

No job is running. Just installed all the components and found alerts of those connection failures on Ambari web interface cluster monitoring.

OS is centos 7, firewalld and selinux are off.

Anyone knows why or how to fix this?

Thanks in advance!

Shannon

Re: App timeline server failed to start due to webUI connection failed

Luos — Fri, 01 Apr 2016 19:49:03 GMT

@Emil

Hi Emil,

Thanks for replying. Sorry I was out of office. Found error in yarn-yarn-timelineserver-xxx.log only (see below). It seems to be missing a timeline plugin related to tez. Ambari used is the newest as of March 2016 and should not have any patch issue related to this, should it? Tez was not running as hive was not used. Do you know how this can be fixed?

Thanks!

Shannon

--------------copied from log---------

2016-03-10 20:57:02,239 INFO timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:loadPlugIns(217)) - Error loading plugin org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl org.apache.hadoop.yarn.exceptions.YarnException: Unable to load class: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:187) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:214) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:183) ... 8 more

2016-03-10 20:57:02,245 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(171)) - Error starting ApplicationHistoryServer java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:223) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)

Re: App timeline server failed to start due to webUI connection failed

Luos — Thu, 07 Apr 2016 21:52:04 GMT

@Emil

Hi Emil,

Turn on Hive with Tez as execution engine. Still same problem. Where should I get this missing plugin, do you know?

Thanks!

Shannon

Re: App timeline server failed to start due to webUI connection failed

qiwang — Tue, 12 Apr 2016 00:41:14 GMT

@shannon luo

I am a solution engineer located in King of Prussia and trying to reproduce the problem you have. Could you please provide some more information about the environment of you cluster

public cloud/private cloud/vm
kernal version of centos7
Java version
node layout for where all the services you have
- Ambari server
- name nodes
- resource manager nodes
- hive/tez/hbase/spark/..
- clients

I assume you following the instruction on the following page during you installation and have prepared the environments following the instruction.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Installing_HDP_AMB/content/ch_Getting_Ready.html

Thanks,

Re: App timeline server failed to start due to webUI connection failed

Luos — Tue, 12 Apr 2016 07:34:31 GMT

@Qi Wang

Hi Qi,

would you please email me at sluo(at)anexinet.com? I can send environment details to your email.

Thanks!

Shannon

Re: App timeline server failed to start due to webUI connection failed

Luos — Fri, 15 Apr 2016 02:41:41 GMT

Thanks to help from @Qi Wang, the problem is solved by moving the appTimeline server to the same server where history server is. Thank you, Qi!

Re: App timeline server failed to start due to webUI connection failed

mayur_1286 — Tue, 03 May 2016 05:15:23 GMT

@qi

@Qi Wang I had a similar problem with the yarn resource manager running on master 2 not being able to connect to yarn App timeline server running on master 3. I moved yarnApp timeline server to master 2 and it works fine now. But just wanted to check does Apptimeline server have to be on the same node as as yarn rm or How are the three components related App timeline server, YARN RM and History server ?

Error logs

resource_management.core.exceptions.Fail: Execution of 'yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000' returned 2. 16/05/02 15:44:15 INFO impl.TimelineClientImpl: Timeline service address: http://master3.com:8188/ws/v1/timeline/

Re: App timeline server failed to start due to webUI connection failed

smilessosweet — Tue, 11 Jul 2017 16:10:34 GMT

Hi, I am new to HDP and I am encountering the same error.

May I know where do I see the history server is?

Could you list down the steps you used to "move the appTimeline server to the same server where the history server is"?

Thank you!

question Re: App timeline server failed to start due to webUI connection failed​ in Support Questions

App timeline server failed to start due to webUI connection failed​

Re: App timeline server failed to start due to webUI connection failed​

Re: App timeline server failed to start due to webUI connection failed​

Re: App timeline server failed to start due to webUI connection failed​

Re: App timeline server failed to start due to webUI connection failed​

Re: App timeline server failed to start due to webUI connection failed​

Re: App timeline server failed to start due to webUI connection failed​

Re: App timeline server failed to start due to webUI connection failed​

question Re: App timeline server failed to start due to webUI connection failed in Support Questions

App timeline server failed to start due to webUI connection failed

Re: App timeline server failed to start due to webUI connection failed

Re: App timeline server failed to start due to webUI connection failed

Re: App timeline server failed to start due to webUI connection failed

Re: App timeline server failed to start due to webUI connection failed

Re: App timeline server failed to start due to webUI connection failed

Re: App timeline server failed to start due to webUI connection failed

Re: App timeline server failed to start due to webUI connection failed