Skip to main content
Da_01
Da_01
Gold ⭐️
July 18, 2019
Solved

How to increase database pool for Director?

Where would someone increase the database connection pool for Director? Would this be in application.properties or in /etc/default/cloudera-director-server?

 

I've noticed after tuning the amount of threads for tasks and the amount of instances in parallel in the application.properties that if I try to provision 4 clusters of 100 nodes each at around the same time or with a small delay that Director has issues with not being able to get database connections causing failures.

 

To pre-empt this I have migrated to a serverless aurora with a minimum of 8 units which from the cloudwatch charts shows no spikes in connections yet Director still fails pipeline errors with the following:

 

 

java.util.concurrent.ExecutionException: org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection

 

I guess what I'm trying to determine is if a larger initial connection pool would help this.

jadair
jadairAnswer
Gold ⭐️⭐️
July 19, 2019

According to the comments on the Jira, the properties will not include hikari. That is consistent with the recommendation in the previously linked stackoverflow thread.

View in thread

7 replies

jadair
Gold ⭐️⭐️
jadair
Gold ⭐️⭐️
July 19, 2019

I can't confirm that your problem is caused by the connection pool size, and have never adjusted the connection pool settings manually, but Director uses spring-boot so you should be able to set the appropriate properties in application.properties as described in this stackoverflow question:
https://stackoverflow.com/questions/25573034/spring-boot-how-do-i-set-jdbc-pool-properties-like-maximum-number-of-connection

 

The property prefix should be spring.datasource, and the supported properties should be these:
http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes

 

For the pool size specifically, you should be able to set it like this:

spring.datasource.maxActive=200

 

Translate
Da_01
Da_01Author
Gold ⭐️
July 19, 2019

Hi Jadair,

 

Thanks for responding and pointing me at the spring configurations for the datasource.

 

However, I tried the following configuration which should keep 100 connections open at all times:

 

application_properties configuration.png

 

 

And observed in the cloudwatch monitoring AWS UI that director's connections when shut off dropped the total database connections from 149 to 139 (note that this database is shared by the CM that director spins up which accounts for the other 130ish connections).

 

When Director was started with the above spring configurations you can see the line return to 149 which indicates that Director only opened 10 connections to the database.

 

This implies that the defaults for spring.datasource.minIdle and spring.datasource.initialSize are still being used which are 10. Defaults are listed from the link you mentioned: http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html#Common_Attributes

 

director_serverless.png

 

 So it seems Director is not respecting the spring.datasource.* configurations from application.properties. As I don't have access to the source code, is Director loading these things from configurations or hardcoding them somewhere? Apologies in advance if any of my assertions are false.

 

Quick addition is I also tried adding the -Dkey=value pairs to the JAVA_OPTS in /etc/default/cloudera-director-server but this didn't work either.

Translate
jadair
Gold ⭐️⭐️
jadair
Gold ⭐️⭐️
July 19, 2019

We're relying on springboot, and are not doing anything special. I notice that in the application.properties embedded in the jar, some of the documented tomcat properties, like testOnBorrow and validationQuery, are being set with a prefix of spring.datasource.tomcat. Based on that stackoverflow answer, I wouldn't expect that to be required for maxActive, but give it a try and let me know if it works.

Translate
Terms & ConditionsAccessibility statement