Created 03-22-2016 09:12 PM
Hello,
I have 5 node cluster using HDP2.4 and Ambari 2.2. Ambari server and App timeline server is on node1, Yarn Resource Manager is on node 3. App timeline server, Resource Manager and Node Manager fail because connection to their webUI port 8188, 8088, 8042 respectively failed. Node 1 has Ambari server on port 8080, which runs fine. But App timeline server 8188 on the same node can not connect.
No job is running. Just installed all the components and found alerts of those connection failures on Ambari web interface cluster monitoring.
OS is centos 7, firewalld and selinux are off.
Anyone knows why or how to fix this?
Thanks in advance!
Shannon
Created 04-14-2016 07:41 PM
Thanks to help from @Qi Wang, the problem is solved by moving the appTimeline server to the same server where history server is. Thank you, Qi!
Created 04-01-2016 12:49 PM
Hi Emil,
Thanks for replying. Sorry I was out of office. Found error in yarn-yarn-timelineserver-xxx.log only (see below). It seems to be missing a timeline plugin related to tez. Ambari used is the newest as of March 2016 and should not have any patch issue related to this, should it? Tez was not running as hive was not used. Do you know how this can be fixed?
Thanks!
Shannon
--------------copied from log---------
2016-03-10 20:57:02,239 INFO timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:loadPlugIns(217)) - Error loading plugin org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl org.apache.hadoop.yarn.exceptions.YarnException: Unable to load class: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:187) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:214) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:183) ... 8 more
2016-03-10 20:57:02,245 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(171)) - Error starting ApplicationHistoryServer java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:223) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)
Created 04-07-2016 02:52 PM
Turn on Hive with Tez as execution engine. Still same problem. Where should I get this missing plugin, do you know?
Thanks!
Shannon
Created 04-11-2016 05:41 PM
I am a solution engineer located in King of Prussia and trying to reproduce the problem you have. Could you please provide some more information about the environment of you cluster
I assume you following the instruction on the following page during you installation and have prepared the environments following the instruction.
Thanks,
Qi
Created 04-12-2016 12:34 AM
Hi Qi,
would you please email me at sluo(at)anexinet.com? I can send environment details to your email.
Thanks!
Shannon
Created 04-14-2016 07:41 PM
Thanks to help from @Qi Wang, the problem is solved by moving the appTimeline server to the same server where history server is. Thank you, Qi!
Created 07-11-2017 09:10 AM
Hi, I am new to HDP and I am encountering the same error.
May I know where do I see the history server is?
Could you list down the steps you used to "move the appTimeline server to the same server where the history server is"?
Thank you!
Created 05-02-2016 10:15 PM
@qi
@Qi Wang I had a similar problem with the yarn resource manager running on master 2 not being able to connect to yarn App timeline server running on master 3. I moved yarnApp timeline server to master 2 and it works fine now. But just wanted to check does Apptimeline server have to be on the same node as as yarn rm or How are the three components related App timeline server, YARN RM and History server ?
Error logs
resource_management.core.exceptions.Fail: Execution of 'yarn org.apache.hadoop.yarn.applications.distributedshell.Client -shell_command ls -num_containers 1 -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -timeout 300000' returned 2. 16/05/02 15:44:15 INFO impl.TimelineClientImpl: Timeline service address: http://master3.com:8188/ws/v1/timeline/