question Re: Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation" in Support Questions

Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation"

ldylag — Tue, 21 Apr 2026 06:30:48 GMT

Hi,

we installed a CDP Private Cloud 7.1.9 cluster (CM 7.11.3) on AWS ec2 machines with Postgres RDS as backend database.

Hive Metastore Server is on two machines:

ap-cdpclusteru1c (role group: Hive Metastore Server Default Group)
ap-cdpclusteru2c. (role group: hive-HIVEMETASTORE)

Configuration in CM looks identical for both Roles

ap-cdpclusteru1c starts successfully but ap-cdpclusteru2c fails to start with following error:

+ exec /cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/lib/hive/bin/hive --config /var/run/cloudera-scm-agent/process/1546344708-hive-HIVEMETASTORE --service metastore -p 9083 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] org.apache.thrift.transport.TTransportException: Failed to start token manager at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:116) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:10853) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:10793) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Caused by: java.io.IOException: Failed to initialize master key at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:215) at org.apache.hadoop.hive.metastore.security.MetastoreDelegationTokenManager.startDelegationTokenSecretManager(MetastoreDelegationTokenManager.java:90) at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:112) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213) ... 10 more Caused by: org.apache.hadoop.hive.metastore.security.DelegationTokenStore$TokenStoreException: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:159) at org.apache.hadoop.hive.metastore.security.DBTokenStore.addMasterKey(DBTokenStore.java:42) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:195) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:357) ... 15 more Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:671) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217) at org.postgresql.Driver.makeConnection(Driver.java:458) at org.postgresql.Driver.connect(Driver.java:260) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72) at org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:90) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.initPMF(PersistenceManagerProvider.java:277) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.lambda$updatePmfProperties$0(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.retry(PersistenceManagerProvider.java:563) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:355) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:850) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:818) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:812) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:142) ... 18 more Caused by: java.lang.ClassNotFoundException: com.ongres.scram.common.stringprep.StringPreparation at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) ... 51 more 24/05/14 17:16:00 ERROR metastore.HiveMetaStore: [main]: org.apache.thrift.transport.TTransportException: Failed to start token manager at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:116) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:10853) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:10793) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.util.RunJar.run(RunJar.java:318) at org.apache.hadoop.util.RunJar.main(RunJar.java:232) Caused by: java.io.IOException: Failed to initialize master key at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:215) at org.apache.hadoop.hive.metastore.security.MetastoreDelegationTokenManager.startDelegationTokenSecretManager(MetastoreDelegationTokenManager.java:90) at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:112) ... 8 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213) ... 10 more Caused by: org.apache.hadoop.hive.metastore.security.DelegationTokenStore$TokenStoreException: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:159) at org.apache.hadoop.hive.metastore.security.DBTokenStore.addMasterKey(DBTokenStore.java:42) at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:195) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:357) ... 15 more Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:671) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217) at org.postgresql.Driver.makeConnection(Driver.java:458) at org.postgresql.Driver.connect(Driver.java:260) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101) at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356) at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199) at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444) at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515) at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72) at org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:90) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.initPMF(PersistenceManagerProvider.java:277) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.lambda$updatePmfProperties$0(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.retry(PersistenceManagerProvider.java:563) at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:253) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:355) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59) at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:850) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:818) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:812) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:142) ... 18 more Caused by: java.lang.ClassNotFoundException: com.ongres.scram.common.stringprep.StringPreparation at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) ... 51 more

Full log attached.

I scanned the jars to search for StringPreparation.class and it has the same results on both machines:

jars$ ls *.jar | while read jar ; do unzip -l $jar | sed -e 's@^@'$jar' @' | grep '\.class$' ; done | awk '/StringPreparation.class/ { print $1, $NF }' config-service-1.0.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class config-service.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class postgresql-42.5.1.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class storage-tool-0.10.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class storage-tool-2.3.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class

Looks like "com/ongres/scram/common/stringprep/StringPreparation.class" is not present anywhere, only the "org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class" is present.

There is RH8 and Java 11 on both machines:

ap-cdpclusteru2c ~$ java --version
openjdk 11.0.22 2024-01-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode)

How can I fix the issue causing this error?

Re: Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation"

DianaTorres — Wed, 15 May 2024 19:49:56 GMT

@ldylag Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Hive experts @cravani @mszurap @james_jones who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation"

mszurap — Thu, 16 May 2024 10:10:33 GMT

Hello @ldylag,

The stacktrace shows that the "org.postgresql.core.v3.ConnectionFactoryImpl" class depends on the "com/ongres/scram/common/stringprep/StringPreparation.class" - and that is not available on the classpath.

By default HMS searches for the JDBC driver under the /usr/share/java. Please compare if the same JDBC driver is available on both HMS hosts. Probably the best would be to use the latest drivers from:

https://jdbc.postgresql.org/download/

Best regards

Miklos

Re: Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation"

ldylag — Mon, 20 May 2024 10:23:23 GMT

During cluster installation we followed the postgresql jdbc instruction:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cm_postgres_jdbc_installation.html
later, when fixing some issues, we removed the /usr/share/java/postgresql-connector-java.jar on u1c.
It stayed on u2c and turned to be problem.
It is default name, so hive process was picking it up:

+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/java/oracle-connector-java.jar

and trying to use.
When we removed it on u2c, hive process started to to take 'cloudera delivered' jar:

+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.5.1.jar:/usr/share/java/oracle-connector-java.jar

and hive could start successfully.

We compared the jars, and seems that 'bad' jar was in version 42.2, good one was 42.5:

$ unzip _postgresql-connector-java.jar -d bad $ cat bad/META-INF/MANIFEST.MF | grep -i Version Manifest-Version: 1.0 Implementation-Version: 42.2.14 JavaPackages-Version: 42.2.14 Specification-Version: 4.2 $ unzip postgresql-42.5.1.jar -d good $ cat good/META-INF/MANIFEST.MF | grep -i Version Manifest-Version: 1.0 Bundle-ManifestVersion: 2 Bundle-Version: 42.5.1 ... Specification-Version: 4.2

Listing of jar contents confirms that the class was missing in jar that we used.

$ unzip -l _postgresql-connector-java.jar | grep com/ongres/scram/common/stringprep/StringPreparation.class $ unzip -l postgresql-42.5.1.jar | grep com/ongres/scram/common/stringprep/StringPreparation.class 296 02-01-1980 00:00 org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class

Why is this needed on CM host:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cm_postgres_jdbc_installation.html
if we removed it and all works as expected on CM host?

Re: Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation"

mszurap — Tue, 21 May 2024 08:12:09 GMT

Thanks @ldylag for sharing the details, that makes sense.

The CM does not necessarily need the PostgreSQL driver to be installed, of course it's needed only when that database type is configured for CM. The one under the "/opt/cloudera/cm/lib" might be shipped to support the "embedded" database, that might have caused a confusion here (if one HMS is on the CM host), but in general the DB drivers are expected to be under the "/usr/share/java".

Cheers

Miklos