Hive fails to start with "Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation"

[#1]

Hi,

we installed a CDP Private Cloud 7.1.9 cluster (CM 7.11.3) on AWS ec2 machines with Postgres RDS as backend database.

Hive Metastore Server is on two machines:

• ap-cdpclusteru1c (role group: Hive Metastore Server Default Group)
• ap-cdpclusteru2c. (role group: hive-HIVEMETASTORE)

Configuration in CM looks identical for both Roles

ap-cdpclusteru1c starts successfully but ap-cdpclusteru2c fails to start with following error:

+ exec /cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/lib/hive/bin/hive --config /var/run/cloudera-scm-agent/process/1546344708-hive-HIVEMETASTORE --service metastore -p 9083
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/cdh/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p6.51045883/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
org.apache.thrift.transport.TTransportException: Failed to start token manager
	at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:116)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:10853)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:10793)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.io.IOException: Failed to initialize master key
	at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:215)
	at org.apache.hadoop.hive.metastore.security.MetastoreDelegationTokenManager.startDelegationTokenSecretManager(MetastoreDelegationTokenManager.java:90)
	at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:112)
	... 8 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213)
	... 10 more
Caused by: org.apache.hadoop.hive.metastore.security.DelegationTokenStore$TokenStoreException: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation
	at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:159)
	at org.apache.hadoop.hive.metastore.security.DBTokenStore.addMasterKey(DBTokenStore.java:42)
	at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:195)
	at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:357)
	... 15 more
Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:671)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217)
	at org.postgresql.Driver.makeConnection(Driver.java:458)
	at org.postgresql.Driver.connect(Driver.java:260)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
	at org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:90)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.initPMF(PersistenceManagerProvider.java:277)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.lambda$updatePmfProperties$0(PersistenceManagerProvider.java:253)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.retry(PersistenceManagerProvider.java:563)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:253)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:355)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:850)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:818)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:812)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:142)
	... 18 more
Caused by: java.lang.ClassNotFoundException: com.ongres.scram.common.stringprep.StringPreparation
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	... 51 more
24/05/14 17:16:00 ERROR metastore.HiveMetaStore: [main]: org.apache.thrift.transport.TTransportException: Failed to start token manager
	at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:116)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:10853)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:10793)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.io.IOException: Failed to initialize master key
	at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:215)
	at org.apache.hadoop.hive.metastore.security.MetastoreDelegationTokenManager.startDelegationTokenSecretManager(MetastoreDelegationTokenManager.java:90)
	at org.apache.hadoop.hive.metastore.AuthFactory.<init>(AuthFactory.java:112)
	... 8 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213)
	... 10 more
Caused by: org.apache.hadoop.hive.metastore.security.DelegationTokenStore$TokenStoreException: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation
	at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:159)
	at org.apache.hadoop.hive.metastore.security.DBTokenStore.addMasterKey(DBTokenStore.java:42)
	at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:195)
	at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:357)
	... 15 more
Caused by: java.lang.NoClassDefFoundError: com/ongres/scram/common/stringprep/StringPreparation
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:671)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217)
	at org.postgresql.Driver.makeConnection(Driver.java:458)
	at org.postgresql.Driver.connect(Driver.java:260)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
	at org.apache.hadoop.hive.metastore.datasource.HikariCPDataSourceProvider.create(HikariCPDataSourceProvider.java:90)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.initPMF(PersistenceManagerProvider.java:277)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.lambda$updatePmfProperties$0(PersistenceManagerProvider.java:253)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.retry(PersistenceManagerProvider.java:563)
	at org.apache.hadoop.hive.metastore.PersistenceManagerProvider.updatePmfProperties(PersistenceManagerProvider.java:253)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:355)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:850)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:818)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:812)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.hadoop.hive.metastore.security.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:142)
	... 18 more
Caused by: java.lang.ClassNotFoundException: com.ongres.scram.common.stringprep.StringPreparation
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	... 51 more

Full log attached.

I scanned the jars to search for StringPreparation.class and it has the same results on both machines:

jars$ ls *.jar | while read jar ; do unzip -l $jar | sed -e 's@^@'$jar' @' | grep '\.class$' ; done | awk '/StringPreparation.class/ { print $1, $NF }'
config-service-1.0.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class
config-service.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class
postgresql-42.5.1.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class
storage-tool-0.10.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class
storage-tool-2.3.0.7.1.9.6-3.jar org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class

Looks like "com/ongres/scram/common/stringprep/StringPreparation.class" is not present anywhere, only the "org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class" is present.

There is RH8 and Java 11 on both machines:

ap-cdpclusteru2c ~$ java --version
openjdk 11.0.22 2024-01-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode)

How can I fix the issue causing this error?

[#2]

@ldylag Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Hive experts @cravani @mszurap @james_jones  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3 · Accepted solution]

Hello @ldylag,

The stacktrace shows that the "org.postgresql.core.v3.ConnectionFactoryImpl" class depends on the "com/ongres/scram/common/stringprep/StringPreparation.class" - and that is not available on the classpath.

By default HMS searches for the JDBC driver under the /usr/share/java. Please compare if the same JDBC driver is available on both HMS hosts. Probably the best would be to use the latest drivers from:

https://jdbc.postgresql.org/download/

Best regards

 Miklos

[#4]

During cluster installation we followed the postgresql jdbc instruction:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cm_postgres_jdbc_installa...
later, when fixing some issues, we removed the /usr/share/java/postgresql-connector-java.jar on u1c.
It stayed on u2c and turned to be problem.
It is default name, so hive process was picking it up:

+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/java/oracle-connector-java.jar

and trying to use.
When we removed it on u2c, hive process started to to take 'cloudera delivered' jar:

+ JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.5.1.jar:/usr/share/java/oracle-connector-java.jar

and hive could start successfully.

We compared the jars, and seems that 'bad' jar was in version 42.2, good one was 42.5:

$ unzip _postgresql-connector-java.jar -d bad
$ cat bad/META-INF/MANIFEST.MF | grep -i Version
Manifest-Version: 1.0
Implementation-Version: 42.2.14
JavaPackages-Version: 42.2.14
Specification-Version: 4.2

$ unzip postgresql-42.5.1.jar -d good
$ cat good/META-INF/MANIFEST.MF | grep -i Version
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Version: 42.5.1
...
Specification-Version: 4.2

Listing of jar contents confirms that the class was missing in jar that we used.

$ unzip -l _postgresql-connector-java.jar | grep com/ongres/scram/common/stringprep/StringPreparation.class
$ unzip -l postgresql-42.5.1.jar | grep com/ongres/scram/common/stringprep/StringPreparation.class
296 02-01-1980 00:00 org/postgresql/shaded/com/ongres/scram/common/stringprep/StringPreparation.class

Why is this needed on CM host:
https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/installation/topics/cm_postgres_jdbc_installa...
if we removed it and all works as expected on CM host?

[#5]

Thanks @ldylag for sharing the details, that makes sense.

The CM does not necessarily need the PostgreSQL driver to be installed, of course it's needed only when that database type is configured for CM. The one under the "/opt/cloudera/cm/lib" might be shipped to support the "embedded" database, that might have caused a confusion here (if one HMS is on the CM host), but in general the DB drivers are expected to be under the "/usr/share/java".

Cheers

 Miklos