Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to start LLAP Daemon after upgrading ambari

Failed to start LLAP Daemon after upgrading ambari

Explorer

After upgrading ambari (agent, server and metrics collector) from 2.6.4 to 2.7.3, i could not start llap service.

We haven't upgrade our hdp version yet, so we are using hdp 2.6.4 with ambri 2.7.3. Not sure if this is the root cause.

When starting llap, it threw exception as following:

rg.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Failed to start LLAP Daemon with exception
java.lang.NoClassDefFoundError: org/apache/commons/configuration2/convert/ListDelimiterHandler
    at java.lang.Class.forName0(Native Method) ~[?:1.8.0_112]
    at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_112]
    at org.apache.hadoop.metrics2.impl.MetricsConfig.getPlugin(MetricsConfig.java:196) ~[hadoop-common-2.7.3.2.6.4.0-91.jar:?]
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.newSink(MetricsSystemImpl.java:529) ~[hadoop-common-2.7.3.2.6.4.0-91.jar:?]
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configureSinks(MetricsSystemImpl.java:501) ~[hadoop-common-2.7.3.2.6.4.0-91.jar:?]
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:480) ~[hadoop-common-2.7.3.2.6.4.0-91.jar:?]
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:189) ~[hadoop-common-2.7.3.2.6.4.0-91.jar:?]
    at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:164) ~[hadoop-common-2.7.3.2.6.4.0-91.jar:?]
    at org.apache.hadoop.hive.llap.metrics.LlapMetricsSystem.initialize(LlapMetricsSystem.java:41) ~[hive-exec-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
    at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.<init>(LlapDaemon.java:240) ~[hive-llap-server-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
    at org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon.main(LlapDaemon.java:529) [hive-llap-server-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration2.convert.ListDelimiterHandler
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_112]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_112]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_112]
4 REPLIES 4
Highlighted

Re: Failed to start LLAP Daemon after upgrading ambari

Super Mentor

@sanyun di

Please refer to the following Note from Docs: https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/ambari_upgr...

Ambari 2.7.3 only supports fully managing a HDP 3.1.x cluster. Not all cluster management operations are supported when using Ambari 2.7.3 with HDP 2.6. Please see Changes to Services and Views for limitations.

.

Also please have a look at : https://supportmatrix.hortonworks.com/

And then click on "Ambari 2.7.3" then you will find that it can manage HDP 3.x only (not HDP 2.x)


Highlighted

Re: Failed to start LLAP Daemon after upgrading ambari

Explorer

Thanks for the suggestion!

For this specific case, we found the reason finally. After upgrading Ambari, we also upgraded Ambari metrics. The new metrics libraries don't contain the required class. After rolling back Ambari metrics, LLAP is fine.

Yes, we should follow the doc, upgrade HDP before Ambari metrics.

Highlighted

Re: Failed to start LLAP Daemon after upgrading ambari

Super Mentor

@sanyun di

Some of the JARs that are shipped with HDP 3.x contains the missing class "org.apache.commons.configuration2.convert.ListDelimiterHandler"

Example:

/usr/hdp/3.0.0.0-1634/hive/jdbc/hive-jdbc-3.1.0.3.0.0.0-1634-standalone.jar
/usr/hdp/3.0.0.0-1634/hive/lib/ranger-hive-plugin-impl/commons-configuration2-2.1.1.jar
/usr/hdp/3.0.0.0-1634/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar


Highlighted

Re: Failed to start LLAP Daemon after upgrading ambari

Explorer

Thanks for the suggestion.

For this specific case, we found the reason finally. After upgrading Ambari, we also upgraded Ambari Metrics. There is no required class in the new metrics libraries. Then we rolled back Ambari Metrics, and LLAP was up.

Yes, we should follow the doc, upgrading HDP before Ambari Metrics.

Don't have an account?
Coming from Hortonworks? Activate your account here