HDP 3 upgrade from 2.6 / metrics collector java class issues

NateRutland — Wed, 22 Aug 2018 00:55:10 GMT

Having some issues getting the Ambari metrics HBase master process / collector process to start. Leaning towards trying to add to classpath for zookeeper? Everything else is working ok.

2018-08-20 21:32:26,397 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZooKeeperCluster and ran successful 'stat' on client port=61181 2018-08-20 21:32:26,397 INFO [main] master.HMasterCommandLine: Starting up instance of localHBaseCluster; master=1, regionserversCount=1 2018-08-20 21:32:26,869 INFO [main] regionserver.RSRpcServices: master/nodex/192.168.x.x:0 server-side HConnection retries=350 2018-08-20 21:32:27,078 INFO [main] ipc.SimpleRpcScheduler: Using fifo as user call queue, count=3 2018-08-20 21:32:27,204 INFO [main] ipc.RpcServer: master/nodex/192.168.x.x:0: started 10 reader(s). 2018-08-20 21:32:27,276 INFO [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties 2018-08-20 21:32:27,325 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMasterorg.apache.commons.configuration2.convert.ListDelimiterHandler at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:143) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217) at org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:153) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:226) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2838) Caused by: java.lang.NoClassDefFoundError: org/apache/commons/configuration2/convert/ListDelimiterHandler at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:196) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:529) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:501) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:480) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:189) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:164) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50) at org.apache.hadoop.hbase.metrics.BaseSourceImpl$DefaultMetricsSystemInitializer.init(BaseSourceImpl.java:48) at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:71) at org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceImpl.<init>(MetricsHBaseServerSourceImpl.java:64) at org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceFactoryImpl.getSource(MetricsHBaseServerSourceFactoryImpl.java:48) at org.apache.hadoop.hbase.ipc.MetricsHBaseServerSourceFactoryImpl.create(MetricsHBaseServerSourceFactoryImpl.java:38) at org.apache.hadoop.hbase.ipc.MetricsHBaseServer.<init>(MetricsHBaseServer.java:39) at org.apache.hadoop.hbase.ipc.RpcServer.<init>(RpcServer.java:2013) at org.apache.hadoop.hbase.regionserver.RSRpcServices.<init>(RSRpcServices.java:921) at org.apache.hadoop.hbase.master.MasterRpcServices.<init>(MasterRpcServices.java:248) at org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:595) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:557) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:425) at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.<init>(HMasterCommandLine.java:309) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139) ... 7 more Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration2.convert.ListDelimiterHandler at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Re: HDP 3 upgrade from 2.6 / metrics collector java class issues

apappu — Wed, 22 Aug 2018 01:15:23 GMT

@Nathan Rutland

Did you apply post installation tasks? can you run "rpm -qa | grep metrics" and see if those are pointing to 2.7 (ambari) version ? if not you will have to do https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_ambari-upgrade/content/upgrade_ambari_metrics.html

Re: HDP 3 upgrade from 2.6 / metrics collector java class issues

NateRutland — Wed, 22 Aug 2018 01:58:20 GMT

@amarnath reddy pappu

I thought I updated everything on that node. The other 2 rpm's were 2.7 but the collector was 2.6. Did the yum upgrade and all is well. Thanks for your assistance. Very apprecitated. I'm going to snapshot all of the VMs now.

Re: HDP 3 upgrade from 2.6 / metrics collector java class issues

apappu — Wed, 22 Aug 2018 02:01:44 GMT

@Nathan Rutland thanks for confirming that it has helped. Can you please mark this as correct answer.

question HDP 3 upgrade from 2.6 / metrics collector java class issues in Archives of Support Questions (Read Only)

HDP 3 upgrade from 2.6 / metrics collector java class issues

Re: HDP 3 upgrade from 2.6 / metrics collector java class issues

Re: HDP 3 upgrade from 2.6 / metrics collector java class issues

Re: HDP 3 upgrade from 2.6 / metrics collector java class issues