Reply
Highlighted
Explorer
Posts: 24
Registered: ‎01-07-2016

Metastore process forgets the password to the backing DB

We are using CDH5.5.1 with kerberos enabled . Every 5-10th time we restart the Hive group (including the metastore) in Cloudera Manager the metastore fails to access the metastore DB. It seems like the password is forgotten. We've seen this both when the DB is postgres on Amazon RDS and when using a local postgres (as provided by the cloudera-manager-server-db-2.x86_64 package).

 

This started to happen after upgrading to CDH5.5.1 from CDH5.3.x. One change in the environment that seems related is the introduction of the hadoop credential provider to store the password.

 

Anybody else that has experienced this issue?

 

Config:

<property>
    <name>hadoop.security.credential.provider.path</name>
    <value>localjceks://file//run/cloudera-scm-agent/process/352-hive-HIVEMETASTORE/creds.localjceks</value>
</property>

Error:

2016-02-29 10:45:08,893 ERROR DataNucleus.Datastore.Schema: [main]: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:postgresql://x.y.z.com:5432/metastore, username = hive. Terminating connection pool (set lazyInit to true if you expect 
to start your database after your app). Original Exception: ------
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "hive"
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
        at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
        at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
        at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
        at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
        at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
        at org.postgresql.Driver.makeConnection(Driver.java:393)
        at org.postgresql.Driver.connect(Driver.java:267)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:187)
        at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
        ...
Announcements