We are using CM API to provision hosts on demand. I have noticed that when we provision multiple hosts at the same time, when we call the "apply_host_template" API call, we run into deadlock issues. I have searched online and through forums, found some references to postgres running out of disk space, which is not the case in our environment.   Here is some info from our environment: Version: Cloudera Express 5.4.1 (#197 built by jenkins on 20150509-0041 git: 003e06d761f80834d39c3a42431a266f0aaee736)   postgres (PostgreSQL) 8.4.20 (We are using the embeded Postgres DB via the cloudera-manager-db. I know its not recommended though unclear exactly why since it's running a real version of postgres) We typically have about 100 hosts in our cluster.   Exception returned from the API: Exception occurred: Cannot commit in rollback only mode. None   Exception in Cloudera Manager Server: 2016-06-14 03:41:04,121 ERROR 583972919@scm-web-3788:com.cloudera.server.cmf.OperationsManagerSubject: Caught exception while processing role update handlers javax.persistence.OptimisticLockException: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [com.cloudera.cmf.model.DbRoleConfigGroup#110] at org.hibernate.ejb.AbstractEntityManagerImpl.wrapStaleStateException(AbstractEntityManagerImpl.java:1416)   Error from Postgres: ERROR: deadlock detected DETAIL: Process 28424 waits for ShareLock on transaction 5213625; blocked by process 10493. Process 10493 waits for ExclusiveLock on tuple (3,28) of relation 17068 of database 16385; blocked by process 28424. Process 28424: update ROLE_CONFIG_GROUPS set OPTIMISTIC_LOCK_VERSION=$1, SERVICE_ID=$2, ROLE_TYPE=$3, NAME=$4, DISPLAY_NAME=$5, BASE=$6 where ROLE_CONFIG_GROUP_ID=$7 and OPTIMISTIC_LOCK_VERSION=$8 Process 10493: update ROLE_CONFIG_GROUPS set OPTIMISTIC_LOCK_VERSION=$1, SERVICE_ID=$2, ROLE_TYPE=$3, NAME=$4, DISPLAY_NAME=$5, BASE=$6 where ROLE_CONFIG_GROUP_ID=$7 and OPTIMISTIC_LOCK_VERSION=$8 HINT: See server log for query details. STATEMENT: update ROLE_CONFIG_GROUPS set OPTIMISTIC_LOCK_VERSION=$1, SERVICE_ID=$2, ROLE_TYPE=$3, NAME=$4, DISPLAY_NAME=$5, BASE=$6 where ROLE_CONFIG_GROUP_ID=$7 and OPTIMISTIC_LOCK_VERSION=$8 ERROR: current transaction is aborted, commands ignored until end of transaction block ... View more

Dear Impala users,   We are seeing the following messages in our catalogd logs:   E0523 05:28:25.921648 15916 catalog-server.cc:279] OutOfMemoryError: Java heap space   Impala version: Impala Shell v2.2.0-cdh5 (2ffd73a) built on Tue Apr 21 12:09:21 PDT 2015   I have tried to increase catalogd memory via CM's catalogd safety valves:   "Catalog Server Command Line Argument Advanced Configuration Snippet (Safety Valve)" set to  -rm_default_memory=8G   And through  "Impala Catalog Server Environment Advanced Configuration Snippet (Safety Valve)" set to JAVA_TOOLS_OPTIONS="-Xmx8GB"   I have verified through jmap that neither of the above settings have increased JVM max heap available to catalogd process.   I have also read on a post (https://groups.google.com/a/cloudera.org/forum/#!topic/impala-user/1QKAzAHVIO0) that catalogd supports jvm_args or jvm-args, however our version of impala/catalogd does not seem to support such an option, as below command produces no result:   catalogd --help | grep jvm    I was wondering if anyone has any suggestions or if anyone can provide insight into how to go about troubleshooting this issue.   Thank you in advance for your time.           ... View more