Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation"

Question

Hi,we installed a CDP Private Cloud 7.1.9 cluster (CM 7.11.3) on AWS ec2 machines with Postgres RDS as backend database.Hive Metastore Server is on two machines:ap-cdpclusteru1c (role group: Hive Metastore Server Default Group)ap-cdpclusteru2c. (role group: hive-HIVEMETASTORE)Configuration in CM looks identical for both Rolesap-cdpclusteru1c starts successfully but ap-cdpclusteru2c fails to start with following error: + exec /cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/lib/hive/bin/hive --config /var/run/cloudera-scm-agent/process/1546344708-hive-HIVEMETASTORE --service metastore -p 9083SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]org.apache.thrift.transport.TTransportException: Failed to start token manager at org.apache.hadoop.hive.metastore.AuthFactory.(AuthFactory.java:116) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:10853) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:10793) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232)Caused by: java.io.IOException: Failed to initialize master key at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:215) at org.apache.hadoop.hive.metastore.security.MetastoreDelegationTokenManager.startDelegationTokenSecretManager(MetastoreDelegationTokenManager.java:90) at org.apache.hadoop.hive.metastore.AuthFactory.(AuthFactory.java:112) ... 8 moreCaused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213) ... 10 moreCaused by: org.apache.hadoop.hive.metastore.security.DelegationTokenStore$TokenStoreException: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:159) at org.apache.hadoop.hive.metastore.security.DBTokenStore.addMasterKey(DBTokenStore.java:42) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:195) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:357) ... 15 moreCaused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:671) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.(PgConnection.java:217) at org.postgresql.Driver.makeConnection(Driver.java:458) at org.postgresql.Driver.connect(Driver.java:260) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515) at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:112) at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:72) at org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:90) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.initPMF(PersistenceManagerProvider.java:277) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.lambda$updatePmfProperties$0(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.retry(PersistenceManagerProvider.java:563) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:355) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:850) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:818) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:812) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:142) ... 18 moreCaused by: java.lang.ClassNotFoundException: com.ongres.scram.common.stringprep.StringPreparation at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) ... 51 more24/05/14 17:16:00 ERROR metastore.HiveMetaStore: [main]: org.apache.thrift.transport.TTransportException: Failed to start token manager at org.apache.hadoop.hive.metastore.AuthFactory.(AuthFactory.java:116) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:10853) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:10793) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232)Caused by: java.io.IOException: Failed to initialize master key at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:215) at org.apache.hadoop.hive.metastore.security.MetastoreDelegationTokenManager.startDelegationTokenSecretManager(MetastoreDelegationTokenManager.java:90) at org.apache.hadoop.hive.metastore.AuthFactory.(AuthFactory.java:112) ... 8 moreCaused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213) ... 10 moreCaused by: org.apache.hadoop.hive.metastore.security.DelegationTokenStore$TokenStoreException: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:159) at org.apache.hadoop.hive.metastore.security.DBTokenStore.addMasterKey(DBTokenStore.java:42) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:195) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:357) ... 15 moreCaused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:671) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.(PgConnection.java:217) at org.postgresql.Driver.makeConnection(Driver.java:458) at org.postgresql.Driver.connect(Driver.java:260) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515) at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:112) at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:72) at org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:90) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.initPMF(PersistenceManagerProvider.java:277) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.lambda$updatePmfProperties$0(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.retry(PersistenceManagerProvider.java:563) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:355) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:59) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:850) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:818) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:812) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:142) ... 18 moreCaused by: java.lang.ClassNotFoundException: com.ongres.scram.common.stringprep.StringPreparation at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) ... 51 more Full log attached.I scanned the jars to search for StringPreparation.class and it has the same results on both machines: jars$ ls *.jar | while read jar ; do unzip -l $jar | sed -e 's@^@'$jar' @' | grep '\.class$' ; done | awk '/StringPreparation.class/ { print $1, $NF }'config-service-1.0.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.classconfig-service.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.classpostgresql-42.5.1.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.classstorage-tool-0.10.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.classstorage-tool-2.3.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class Looks like "com/ongres/scram/common/stringprep/StringPreparation.class" is not present anywhere, only the "org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class" is present.There is RH8 and Java 11 on both machines:ap-cdpclusteru2c ~$ java --versionopenjdk 11.0.22 2024-01-16 LTSOpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode)How can I fix the issue causing this error?

mszurap · Accepted Answer

Hello @ldylag,The stacktrace shows that the "org.postgresql.core.v3.ConnectionFactoryImpl" class depends on the "com/ongres/scram/common/stringprep/StringPreparation.class" - and that is not available on the classpath.By default HMS searches for the JDBC driver under the /usr/share/java. Please compare if the same JDBC driver is available on both HMS hosts. Probably the best would be to use the latest drivers from:https://jdbc.postgresql.org/download/Best regards Miklos

ldylag · Answer

During cluster installation we followed the postgresql jdbc instruction:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cm_postgres_jdbc_installation.html
later, when fixing some issues, we removed the /usr/share/java/postgresql-connector-java.jar on u1c.
It stayed on u2c and turned to be problem.
It is default name, so hive process was picking it up:

+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/java/oracle-connector-java.jar

and trying to use.
When we removed it on u2c, hive process started to to take 'cloudera delivered' jar:

+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.5.1.jar:/usr/share/java/oracle-connector-java.jar

and hive could start successfully.

We compared the jars, and seems that 'bad' jar was in version 42.2, good one was 42.5:

$ unzip _postgresql-connector-java.jar -d bad
$ cat bad/META-INF/MANIFEST.MF | grep -i Version
Manifest-Version: 1.0
Implementation-Version: 42.2.14
JavaPackages-Version: 42.2.14
Specification-Version: 4.2

$ unzip postgresql-42.5.1.jar -d good
$ cat good/META-INF/MANIFEST.MF | grep -i Version
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Version: 42.5.1
...
Specification-Version: 4.2

Listing of jar contents confirms that the class was missing in jar that we used.

$ unzip -l _postgresql-connector-java.jar | grep com/ongres/scram/common/stringprep/StringPreparation.class
$ unzip -l postgresql-42.5.1.jar | grep com/ongres/scram/common/stringprep/StringPreparation.class
296 02-01-1980 00:00 org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class

Why is this needed on CM host:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cm_postgres_jdbc_installation.html
if we removed it and all works as expected on CM host?

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.