Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cloudera Manager Database locks

Highlighted

Cloudera Manager Database locks

Explorer

We are using Cloudera Manager API to provision spot nodes in and out of the cluster. We see the following errors in our cloudera manager server logs, and at the point of exceptions, nodes are not able to join the cluster. Our cluster is running CDH 5.7.2. Any help would be appreciated.

 

Cloudera Manager Log:

2017-06-14 14:59:54,599 WARN 236843632@scm-web-3939:com.cloudera.api.ApiExceptionMapper: Unexpected exception.
javax.persistence.RollbackException: Error while committing the transaction
	at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:92)
	at com.cloudera.enterprise.AbstractWrappedEntityManager.commit(AbstractWrappedEntityManager.java:110)
	at com.cloudera.cmf.persist.CmfEntityManager.commit(CmfEntityManager.java:377)
	at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransaction(ManagerDaoBase.java:276)
	at com.cloudera.api.dao.impl.ManagerDaoBase.access$300(ManagerDaoBase.java:83)
	at com.cloudera.api.dao.impl.ManagerDaoBase$TransactionCallable.call(ManagerDaoBase.java:315)
	at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:30)
	at com.cloudera.api.dao.impl.ManagerDaoBase.runInNewTransactionWithRetry(ManagerDaoBase.java:224)
	at com.cloudera.api.dao.impl.ManagerDaoBase.invoke(ManagerDaoBase.java:350)
	at com.sun.proxy.$Proxy134.issueGlobalCommand(Unknown Source)
	at com.cloudera.api.v2.impl.ClouderaManagerResourceV2Impl.hostsStartRolesCommand(ClouderaManagerResourceV2Impl.java:72)
	at sun.reflect.GeneratedMethodAccessor1612.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
	at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
	at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:256)
	at com.cloudera.api.ApiInvoker.invoke(ApiInvoker.java:116)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
	at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
	at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
	at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
	at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
	at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at com.cloudera.api.ApiBasicAuthFilter.doFilter(ApiBasicAuthFilter.java:44)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: javax.persistence.OptimisticLockException
	at org.hibernate.ejb.AbstractEntityManagerImpl.wrapStaleStateException(AbstractEntityManagerImpl.java:1413)
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1329)
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
	at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:80)
	... 83 more
Caused by: org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [com.cloudera.cmf.model.DbService#90]
	at org.hibernate.persister.entity.AbstractEntityPersister.check(AbstractEntityPersister.java:2509)
	at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3228)
	at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3126)
	at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3456)
	at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:140)
	at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:377)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:369)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:287)
	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:339)
	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)
	at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1234)
	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404)
	at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:101)
	at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:175)
	at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:75)
	... 83 more

 

Error we are seeing sent through the API:

Traceback (most recent call last):
  File "/srv/jobs-staging/hadoop-auto-scaler/lib/managed.py", line 13, in decorator
    result = func(*args, **kwargs)
  File "/srv/jobs-staging/hadoop-auto-scaler/lib/cm_api_helper.py", line 58, in start_roles
    result = self.cm.hosts_start_roles([host]).wait()
  File "/usr/lib/python2.7/site-packages/cm_api/endpoints/cms.py", line 240, in hosts_start_roles
    return self._cmd('hostsStartRoles', data=host_names)
  File "/usr/lib/python2.7/site-packages/cm_api/endpoints/types.py", line 331, in _cmd
    data=data, params=params, api_version=api_version)
  File "/usr/lib/python2.7/site-packages/cm_api/endpoints/types.py", line 362, in _post
    api_version)
  File "/usr/lib/python2.7/site-packages/cm_api/endpoints/types.py", line 380, in _call
    api_version)
  File "/usr/lib/python2.7/site-packages/cm_api/endpoints/types.py", line 137, in call
    ret = method(path, data=data, params=params)
  File "/usr/lib/python2.7/site-packages/cm_api/resource.py", line 148, in post
    self._make_headers(contenttype))
  File "/usr/lib/python2.7/site-packages/cm_api/resource.py", line 73, in invoke
    headers=headers)
  File "/usr/lib/python2.7/site-packages/cm_api/http_client.py", line 174, in execute
    raise self._exc_class(ex)
ApiException: Error while committing the transaction (error 500)
2 REPLIES 2

Re: Cloudera Manager Database locks

Cloudera Employee

What's the database you are using? What's the version of Cloudera Manager?

Re: Cloudera Manager Database locks

Explorer

CM About box:

Version: Cloudera Express 5.7.2 (#17 built by jenkins on 20160722-1347 git: 1ac5976e8ad8f16506c2db236aee83141915c44d)

Java VM Name: Java HotSpot(TM) 64-Bit Server VM

Java VM Vendor: Oracle Corporation

Java Version: 1.8.0_101

Server Time: Jun 15, 2017 1:54:09 PM, Eastern Daylight Time (EDT)

 

Database:

AWS RDS PostgreSQL 9.4.7

Instance db.m4.large

 

Thank you for your help!

Don't have an account?
Coming from Hortonworks? Activate your account here