Created 02-12-2021 11:47 PM
Hi, I'm having an issue while upgrading the Cloudera External Postgres database(aws rds) from 9.5 to the latest 12.3. I did the below steps:
1. I'm using AWS RDS POSTGRES for Cloudera external DB. The PSQL 9.5 has end of life. So need to upgrade to the latest version of 12.3. We had to upgrade the copy 9.5 version to the latest 12.3. Both versions of rds psql roles, security, and databases are identical.
2. I've stopped the services that using the embedded database server and also ran the cloudera-scm-server stop command and I have changed the com.cloudera.cmf.db.host to a new version of rds psql.
3. After the above changes, I have started the cloudera-scm-server successfully with command but the UI didn't open, so I have checked the SCM logs then I found the below:
2021-02-13 05:12:07,164 WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1:com.mchange.v2.resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@8326160 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
org.postgresql.util.PSQLException: This ResultSet is closed.
at org.postgresql.jdbc2.AbstractJdbc2ResultSet.checkClosed(AbstractJdbc2ResultSet.java:2674)
at org.postgresql.jdbc2.AbstractJdbc2ResultSet.setFetchSize(AbstractJdbc2ResultSet.java:1791)
at org.postgresql.jdbc4.Jdbc4Statement.createResultSet(Jdbc4Statement.java:39)
at org.postgresql.jdbc2.AbstractJdbc2Statement$StatementResultHandler.handleResultRows(AbstractJdbc2Statement.java:211)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1772)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
at org.postgresql.jdbc2.AbstractJdbc2Connection.execSQLUpdate(AbstractJdbc2Connection.java:263)
at org.postgresql.jdbc2.AbstractJdbc2Connection.getTransactionIsolation(AbstractJdbc2Connection.java:775)
at com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:104)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:198)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
2021-02-13 05:12:07,165 WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0:com.mchange.v2.resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@8d16f8d -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
org.postgresql.util.PSQLException: This ResultSet is closed.
4. I have never seen an error like that but as per my understanding, it is related to the Postgres JDBC jar file-related. I have searched the Postgres jar files related to the old version of 9.5 and found the below:
/usr/share/cmf/cloudera-navigator-server/jars/postgresql-9.0-801.jdbc4.jar
/usr/share/cmf/cloudera-navigator-server/libs/cdh5/postgresql-9.0-801.jdbc4.jar
/usr/share/cmf/common_jars/postgresql-9.0-801.jdbc4.jar
/usr/share/cmf/cloudera-scm-telepub/jars/postgresql-9.0-801.jdbc4.jar
/usr/share/cmf/cloudera-scm-telepub/libs/cdh5/postgresql-9.0-801.jdbc4.jar
/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/jars/postgresql-9.0-801.jdbc4.jar
/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/impala/lib/postgresql-9.0-801.jdbc 4.jar
/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/oozie/libserver/postgresql-9.0-801 .jdbc4.jar
/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/lib/oozie/libtools/postgresql-9.0-801.jdbc4.jar
Can someone help me to replace the old jar version with new version jar files. I know it is not simple to replace files just like that but do I really need to stop the cluster and do configs/ clean/ cache? I still want to keep the current version of CDH 5.13 for now. I have no idea what to do now. Can someone please help with the next steps?
I really appreciate your time and help.
Thanks,
Venkat
Created 02-16-2021 02:59 AM
@gwsolutions This error seems red-herring. You have to see full CM server log file. You can upload here if you want another pair of eye to take a look.
Created 02-17-2021 12:56 PM
As I mentioned, This is related to the psql JDBC driver version issue. Not sure how to replace the psql 9.x driver version with the 12.x driver version without losing the cluster.
Created 02-21-2021 06:28 AM
@gwsolutions Replacing driver is not gonna loose cluster. You just have to replace Jar file if that's compatible.
If needed you can always take DB backup and dump into in fresh DB.