CM API Exception occurred: Cannot commit in rollback only mode. None

[#1]

We are using CM API to provision hosts on demand. I have noticed that when we provision multiple hosts at the same time, when we call the "apply_host_template" API call, we run into deadlock issues. I have searched online and through forums, found some references to postgres running out of disk space, which is not the case in our environment.

Here is some info from our environment:

Version: Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)

postgres (PostgreSQL) 8.4.20 (We are using the embeded Postgres DB via the cloudera-manager-db. I know its not recommended though unclear exactly why since it's running a real version of postgres)

We typically have about 100 hosts in our cluster.

Exception returned from the API:

Exception occurred: Cannot commit in rollback only mode. None

Exception in Cloudera Manager Server:

2016-06-14 03:41:04,121 ERROR 583972919@scm-web-3788:com.cloudera.server.cmf.OperationsManagerSubject: Caught exception while processing role update handlers
javax.persistence.OptimisticLockException: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [com.cloudera.cmf.model.DbRoleConfigGroup#110]
at org.hibernate.ejb.AbstractEntityManagerImpl.wrapStaleStateException(AbstractEntityManagerImpl.java:1416)

Error from Postgres:

ERROR: deadlock detected
DETAIL: Process 28424 waits for ShareLock on transaction 5213625; blocked by process 10493.
Process 10493 waits for ExclusiveLock on tuple (3,28) of relation 17068 of database 16385; blocked by process 28424.
Process 28424: update ROLE_CONFIG_GROUPS set OPTIMISTIC_LOCK_VERSION=$1, SERVICE_ID=$2, ROLE_TYPE=$3, NAME=$4, DISPLAY_NAME=$5, BASE=$6 where ROLE_CONFIG_GROUP_ID=$7 and OPTIMISTIC_LOCK_VERSION=$8
Process 10493: update ROLE_CONFIG_GROUPS set OPTIMISTIC_LOCK_VERSION=$1, SERVICE_ID=$2, ROLE_TYPE=$3, NAME=$4, DISPLAY_NAME=$5, BASE=$6 where ROLE_CONFIG_GROUP_ID=$7 and OPTIMISTIC_LOCK_VERSION=$8
HINT: See server log for query details.
STATEMENT: update ROLE_CONFIG_GROUPS set OPTIMISTIC_LOCK_VERSION=$1, SERVICE_ID=$2, ROLE_TYPE=$3, NAME=$4, DISPLAY_NAME=$5, BASE=$6 where ROLE_CONFIG_GROUP_ID=$7 and OPTIMISTIC_LOCK_VERSION=$8
ERROR: current transaction is aborted, commands ignored until end of transaction block

[#2 · Accepted solution]

We have added several improvements in our latest CM 5.7.x that handles OptimisticLockExceptions more grecefully, please consider upgrading your CM to the latest release.

[#3]

We just upgraded our cloudera manager to 5.7.2 and unfortunately we are still hitting deadlocks, Here is one example: 2016-08-04 16:54:15,829 ERROR 959055168@scm-web-1733:org.hibernate.engine.jdbc.spi.SqlExceptionHelper: ERROR: deadlock detected Detail: Process 16167 waits for ShareLock on transaction 6487968; blocked by process 14243. Process 14243 waits for ExclusiveLock on tuple (0,39) of relation 17068 of database 16385; blocked by process 16167. Hint: See server log for query details.

[#4]

Hi I use sahara to deploy a cluster with CDH 5.4.0, which uses cm-api to provision cluster.  And I got error when I execute concurrent command "deployClientconfig". Can you give me more detail about the fix?

2016-12-01 06:12:26,959 DEBUG 71847465@scm-web-22:com.cloudera.api.ApiInvoker: API Error 500 [/api/v8/clusters/error-of-course/services/hdfs01/commands/deployClientConfig]: ApiErrorMessage{org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [com.cloudera.cmf.model.DbClientConfig#1]}
2016-12-01 06:12:26,962 DEBUG 71847465@scm-web-22:com.cloudera.api.ApiInvoker: API Error 500 [/api/v8/clusters/error-of-course/services/hdfs01/commands/deployClientConfig]: ApiErrorMessage{org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [com.cloudera.cmf.model.DbClientConfig#1]}
2016-12-01 06:12:26,962 DEBUG 71847465@scm-web-22:com.cloudera.api.ApiInvoker: API Error 500 [/api/v8/clusters/error-of-course/services/hdfs01/commands/deployClientConfig]: ApiErrorMessage{org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [com.cloudera.cmf.model.DbClientConfig#1]}
2016-12-01 06:12:26,962 DEBUG 71847465@scm-web-22:com.cloudera.api.ApiInvoker: API Error 500 [/api/v8/clusters/error-of-course/services/hdfs01/commands/deployClientConfig]: ApiErrorMessage{org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [com.cloudera.cmf.model.DbClientConfig#1]}