Ranger Admin Unable to acquire connections to underlying database KmsKeyMgr.svcStore

[#1]

I've upgraded to ambari to 2.7.3.0. In addition to upgrading ambari, I upgraded postgres from v9 to v10. After upgrading postgres, I can no longer start the ranger admin service within ambari. I've been seeing the notification: "Ranger admin service is not reachable, please restart the service." When I try to restart the service I observe the following log in /var/log/ranger/admin/catalina.out (see stack trace below):

Any ideas on what is going on here? I have a feeling there is a conflict between v9/v10 postgres db possibly, but I'm not entirely sure on this.

Dec 02, 2019 2:11:13 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kmsKeyMgr': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: org.apache.ranger.biz.ServiceDBStore org.apache.ranger.biz.KmsKeyMgr.svcStore; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceDBStore': Invocation of init method failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!
Error Code: 0
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:287)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:605)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:472)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:383)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:283)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1572)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1562)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: org.apache.ranger.biz.ServiceDBStore org.apache.ranger.biz.KmsKeyMgr.svcStore; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceDBStore': Invocation of init method failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: Connections could not be acquired from the underlying database!
Error Code: 0

[#2]

@rwinters 

You could be right that your issue is incompatibility? Can you check exactly what version of Postgres you upgraded to?

It's usually advisable to check the Hortonworks support matrix  before launching any upgrade I only hope it's a dev environment  see screenshot

 Here you see PostgreSQL 10.7 is only compatible with Ambari 2.7.4 and HDP 3.1.4  very useful tool.

Happy hadooping 

[#3]

Thanks for your response. Sorry for my the late response, but I wanted to pass this along in case someone runs into this issue. So I did some more testing and what I found is that I had an incompatible postgresql-jdbc driver. So I was upgrading from 9.6 to 10.x and it appears the jdbc driver I was using was pretty old and incompatible with version 10.x. Once I upgraded the postgresql-jdbc driver everything worked just fine. Here is what I did to resolve my issue:

• wget https://jdbc.postgresql.org/download/postgresql-42.2.9.jar
• mv /usr/share/java/postgresql-jdbc.jar /usr/share/java/postgresql-jdbc.jar.old  (backing up the old driver)
• cp ~/postgresql-42.2.9.jar /usr/share/java/postgresql-jdbc.jar
• ambari-server setup --jdbc-db=postgres --jdbc-driver=/usr/share/java/postgresql-jdbc.jar