Updating HDP 2.4 to 2.6 storm nimbus not starting.

[#1]

I upgraded from HDP 2.4 to 2.6 using express upgrade. All services are running fine except Nimbus. I get following error:

java.lang.NoClassDefFoundError: backtype/storm/metric/IClusterReporter
        at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_40]
        at java.lang.ClassLoader.defineClass(ClassLoader.java:760) ~[?:1.8.0_40]
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_40]
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[?:1.8.0_40]
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[?:1.8.0_40]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[?:1.8.0_40]
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[?:1.8.0_40]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_40]
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[?:1.8.0_40]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_40]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_40]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_40]
        at java.lang.Class.forName0(Native Method) ~[?:1.8.0_40]
        at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_40]
        at org.apache.storm.metric.ClusterMetricsConsumerExecutor.prepare(ClusterMetricsConsumerExecutor.java:45) ~[storm-core-1.1.0.2.6.2.0-205.jar:1.1.0.2.6.2.0-205]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_40]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_40]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_40]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_40]
        at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
        at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:313) ~[clojure-1.7.0.jar:?]
        at org.apache.storm.daemon.nimbus$fn__9790$exec_fn__3654__auto____9791.invoke(nimbus.clj:2469) ~[storm-core-1.1.0.2.6.2.0-205.jar:1.1.0.2.6.2.0-205]
        at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]
        at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
        at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
        at org.apache.storm.daemon.nimbus$fn__9790$service_handler__9823.doInvoke(nimbus.clj:2446) ~[storm-core-1.1.0.2.6.2.0-205.jar:1.1.0.2.6.2.0-205]
        at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
        at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2534) ~[storm-core-1.1.0.2.6.2.0-205.jar:1.1.0.2.6.2.0-205]
        at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2567) ~[storm-core-1.1.0.2.6.2.0-205.jar:1.1.0.2.6.2.0-205]
        at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2590) ~[storm-core-1.1.0.2.6.2.0-205.jar:1.1.0.2.6.2.0-205]
        at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
        at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
        at org.apache.storm.daemon.nimbus.main(Unknown Source) ~[storm-core-1.1.0.2.6.2.0-205.jar:1.1.0.2.6.2.0-205]
Caused by: java.lang.ClassNotFoundException: backtype.storm.metric.IClusterReporter
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_40]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_40]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_40]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_40]
        ... 33 more

[#2 · Accepted solution]

@Eon kitex

There is an ambari metrics plugin used by Nimbus that also needed to be upgrade and was provided in the ambari-metrics-monitor or hadoop-sink components.

To perform the upgrade from both ambari-metrics-monitor and ambari-metrics-hadoop-sink on all nodes of the cluster.

If you have metrics service added, it is recommended to shutdown all of the Ambari Metrics components and run the following:

-> yum upgrade ambari-metrics-monitor
-> yum upgrade ambari-metrics-hadoop-sink

on each node of the cluster and then try starting Nimbus service.

[#3]

I had done this in one of the node but forgot to perform this on all of the nodes. It works seemlessly.