App timeline server fails to start after hdp 2.6.5 upgrade and installation of Apache slider with error: ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl

[#1]

Hello All ,

I have upgraded hdp to 2.6.5.0 from hdp 2.5.0.0 and installed Apache slider on hdp 2.6.5.0 .

After installation of apache slider the App timeline server is not starting .Below error is displayed in the logs:

2018-08-22 13:39:08,699 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service EntityGroupFSTimelineStore failed in state INITED; cause: java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:256) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:196) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:243) ... 7 more 2018-08-22 13:39:08,700 INFO timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:serviceStop(330)) - Stopping EntityGroupFSTimelineStore 2018-08-22 13:39:08,702 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer failed in state INITED; cause: java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:256) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:196) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:111) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:174) at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:184) Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197) at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165) at java.lang.Class.forName0(Native Method)

[#2 · Accepted solution]

Is TEZ installed on your cluster? From the "Upgrading Timeline Server 1.0 to 1.5" documentation;

If you have Tez enabled, the tez-client must be installed on the ATS server. You must also perform this additional step:
  <property>
    <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes</name>
    <value>org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl</value>
  </property>

Install the TEZ client on the ATS Server if you haven't got this already.
If you already have the TEZ client installed on the ATS server, we may have to try a reinstall of the TEZ client on that host.

[#3]

@Jonathan Sneep : Thanks ! i didnt had tez client on the ATS server.After installation the ATS came up .

[#4]

Awesome, glad to hear that it's working for you now 🙂