Unable to delete hosts or assign roles

[#1]

Hi,

I'm currently running Cloudera Manager 5.0.2. I am having problem when I try to delete host. And also when I try to assign roles to host.

I added 2 hosts into cluster:

1. Host 1 did not pass host inspection because I installed the wrong CDH version (5.1.2); other hosts are running CDH 5.0.3.

2. Host 2 installed successfully, but I run into error when I try to add role.

The error message is different, but I feel that the 2 problems are related somehow.

Here's the error log that I received when I try to delete host:

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: could not perform addBatch
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()

 Stack Trace:

Caused by: org.hibernate.exception.GenericJDBCException:could not perform addBatch
at StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()

 Stack Trace:

1. StandardSQLExceptionConverter.java line 54
   in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
2. SqlExceptionHelper.java line 125
   in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
3. BatchingBatch.java line 114
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
4. BatchingBatch.java line 101
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
5. AbstractBatchImpl.java line 149
   in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
6. JdbcCoordinatorImpl.java line 198
   in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
7. ActionQueue.java line 372
   in org.hibernate.engine.spi.ActionQueue executeActions()
8. ActionQueue.java line 286
   in org.hibernate.engine.spi.ActionQueue executeActions()
9. AbstractFlushingEventListener.java line 339
   in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
10. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
11. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
12. AbstractEntityManagerImpl.java line 996
    in org.hibernate.ejb.AbstractEntityManagerImpl flush()
13. CmfEntityManager.java line 998
    in com.cloudera.cmf.persist.CmfEntityManager flush()
14. HardwareController.java line 514
    in com.cloudera.server.web.cmf.HardwareController deleteHosts()
15. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
16. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
17. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
18. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
19. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
20. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
21. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
22. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
23. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
24. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
25. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
26. Method.java line 606
    in java.lang.reflect.Method invoke()
27. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
28. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
29. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
30. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
31. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
32. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
33. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
34. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
35. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
36. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
37. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
38. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
39. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
40. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
41. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
42. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
43. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
44. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
45. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
47. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
54. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
55. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
60. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
61. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
62. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
63. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
64. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
65. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
66. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
67. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
68. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
69. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
70. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
71. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
72. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
73. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
74. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
75. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
76. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
77. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
78. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
79. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
80. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
81. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
82. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
83. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
84. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
85. Server.java line 326
    in org.mortbay.jetty.Server handle()
86. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
87. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
88. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
89. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
90. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
91. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
92. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.sql.BatchUpdateException:Batch entry 0 insert into AUDITS (OPTIMISTIC_LOCK_VERSION, ACTING_USER_ID, AUDIT_TYPE, CLUSTER_ID, SERVICE_ID, ROLE_ID, COMMAND_ID, USER_ID, HOST_ID, HOST_TEMPLATE_ID, CONFIG_CONTAINER_ID, CREATED_INSTANT, MESSAGE, IP_ADDRESS, AUDIT_ID) values ('0', '1', 'HOST', NULL, NULL, NULL, NULL, NULL, '4', NULL, NULL, '1409644295347', 'Deleted host analyticpapp6.telkomsel.co.id', '10.2.133.134', '203') was aborted. Call getNextException to see the cause.
at AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()

 Stack Trace:

1. AbstractJdbc2Statement.java line 2598
   in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
2. QueryExecutorImpl.java line 1836
   in org.postgresql.core.v3.QueryExecutorImpl processResults()
3. QueryExecutorImpl.java line 407
   in org.postgresql.core.v3.QueryExecutorImpl execute()
4. AbstractJdbc2Statement.java line 2737
   in org.postgresql.jdbc2.AbstractJdbc2Statement executeBatch()
5. NewProxyPreparedStatement.java line 1723
   in com.mchange.v2.c3p0.impl.NewProxyPreparedStatement executeBatch()
6. BatchingBatch.java line 110
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
7. BatchingBatch.java line 101
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
8. AbstractBatchImpl.java line 149
   in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
9. JdbcCoordinatorImpl.java line 198
   in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
10. ActionQueue.java line 372
    in org.hibernate.engine.spi.ActionQueue executeActions()
11. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
12. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
13. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
14. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
15. AbstractEntityManagerImpl.java line 996
    in org.hibernate.ejb.AbstractEntityManagerImpl flush()
16. CmfEntityManager.java line 998
    in com.cloudera.cmf.persist.CmfEntityManager flush()
17. HardwareController.java line 514
    in com.cloudera.server.web.cmf.HardwareController deleteHosts()
18. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
19. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
20. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
21. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
22. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
23. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
24. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
25. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
26. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
27. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
28. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
29. Method.java line 606
    in java.lang.reflect.Method invoke()
30. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
31. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
32. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
33. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
34. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
35. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
36. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
37. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
38. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
39. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
40. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
41. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
42. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
43. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
44. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
45. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
46. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
47. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
48. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
50. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
60. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
61. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
62. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
63. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
64. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
65. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
66. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
67. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
68. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
69. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
70. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
71. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
72. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
73. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
74. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
75. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
76. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
77. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
78. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
79. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
80. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
81. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
82. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
83. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
84. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
85. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
86. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
87. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
88. Server.java line 326
    in org.mortbay.jetty.Server handle()
89. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
90. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
91. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
92. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
93. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
94. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
95. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

[#2]

Referring to the list, cloudera-service-monitor and cloudera-host-monitor are the two largest. Could this be related to the Time Series Storage set for cloudera-service-monitor at 10 Gb?

I cannot commit any update via cloudera manager at the moment, i tried to lower the retention then I get this error:

java.lang.IllegalStateException:Cannot commit in rollback only mode.
at Preconditions.java line 149
in com.google.common.base.Preconditions checkState()

[#3 · Accepted solution]

I've solved this issue.

Cloudera Embedded DB cannot write because it is located in my /var directory (under /var/lib/cloudera-scm-server-db) which is full. I use this step-by-step solution (credits to puneethabm from google group):

1) Stop the scm service:

# service cloudera-scm-server stop

# service cloudera-scm-server-db stop 

2) Copy the cloudera-scm-server-db, retaining the permissions as below:
#cd /var/lib

#cp -rp cloudera-scm-server-db /dir1/lib/

#cd /var/lib/cloudera-scm-server-db

#rm -rf data/

3) Create symlink
#cd /var/lib/cloudera-scm-server-db

#ln -s /dir1/lib/cloudera-scm-server-db/data data 

4) Start services:

# service cloudera-scm-server-db start

DB initialization done.

waiting for server to start.... done

server started

# service cloudera-scm-server start

By creating symlink to disk with sufficient free space, it allows Cloudera Embedded DB to perform write transaction again.

[#4]

Glad you solved this!

Keep in mind that when using the symlink, you may need to re-create it whenever you upgrade your cloudera-scm-server-db package in the future, since the symlink confuses the packaging code.

Thanks,
Darren