Unable to delete hosts or assign roles

[#1]

Hi,

I'm currently running Cloudera Manager 5.0.2. I am having problem when I try to delete host. And also when I try to assign roles to host.

I added 2 hosts into cluster:

1. Host 1 did not pass host inspection because I installed the wrong CDH version (5.1.2); other hosts are running CDH 5.0.3.

2. Host 2 installed successfully, but I run into error when I try to add role.

The error message is different, but I feel that the 2 problems are related somehow.

Here's the error log that I received when I try to delete host:

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: could not perform addBatch
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()

 Stack Trace:

Caused by: org.hibernate.exception.GenericJDBCException:could not perform addBatch
at StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()

 Stack Trace:

1. StandardSQLExceptionConverter.java line 54
   in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
2. SqlExceptionHelper.java line 125
   in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
3. BatchingBatch.java line 114
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
4. BatchingBatch.java line 101
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
5. AbstractBatchImpl.java line 149
   in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
6. JdbcCoordinatorImpl.java line 198
   in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
7. ActionQueue.java line 372
   in org.hibernate.engine.spi.ActionQueue executeActions()
8. ActionQueue.java line 286
   in org.hibernate.engine.spi.ActionQueue executeActions()
9. AbstractFlushingEventListener.java line 339
   in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
10. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
11. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
12. AbstractEntityManagerImpl.java line 996
    in org.hibernate.ejb.AbstractEntityManagerImpl flush()
13. CmfEntityManager.java line 998
    in com.cloudera.cmf.persist.CmfEntityManager flush()
14. HardwareController.java line 514
    in com.cloudera.server.web.cmf.HardwareController deleteHosts()
15. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
16. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
17. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
18. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
19. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
20. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
21. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
22. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
23. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
24. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
25. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
26. Method.java line 606
    in java.lang.reflect.Method invoke()
27. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
28. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
29. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
30. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
31. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
32. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
33. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
34. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
35. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
36. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
37. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
38. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
39. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
40. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
41. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
42. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
43. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
44. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
45. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
46. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
47. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
54. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
55. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
60. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
61. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
62. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
63. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
64. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
65. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
66. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
67. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
68. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
69. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
70. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
71. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
72. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
73. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
74. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
75. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
76. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
77. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
78. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
79. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
80. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
81. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
82. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
83. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
84. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
85. Server.java line 326
    in org.mortbay.jetty.Server handle()
86. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
87. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
88. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
89. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
90. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
91. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
92. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.sql.BatchUpdateException:Batch entry 0 insert into AUDITS (OPTIMISTIC_LOCK_VERSION, ACTING_USER_ID, AUDIT_TYPE, CLUSTER_ID, SERVICE_ID, ROLE_ID, COMMAND_ID, USER_ID, HOST_ID, HOST_TEMPLATE_ID, CONFIG_CONTAINER_ID, CREATED_INSTANT, MESSAGE, IP_ADDRESS, AUDIT_ID) values ('0', '1', 'HOST', NULL, NULL, NULL, NULL, NULL, '4', NULL, NULL, '1409644295347', 'Deleted host analyticpapp6.telkomsel.co.id', '10.2.133.134', '203') was aborted. Call getNextException to see the cause.
at AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()

 Stack Trace:

1. AbstractJdbc2Statement.java line 2598
   in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
2. QueryExecutorImpl.java line 1836
   in org.postgresql.core.v3.QueryExecutorImpl processResults()
3. QueryExecutorImpl.java line 407
   in org.postgresql.core.v3.QueryExecutorImpl execute()
4. AbstractJdbc2Statement.java line 2737
   in org.postgresql.jdbc2.AbstractJdbc2Statement executeBatch()
5. NewProxyPreparedStatement.java line 1723
   in com.mchange.v2.c3p0.impl.NewProxyPreparedStatement executeBatch()
6. BatchingBatch.java line 110
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
7. BatchingBatch.java line 101
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
8. AbstractBatchImpl.java line 149
   in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
9. JdbcCoordinatorImpl.java line 198
   in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
10. ActionQueue.java line 372
    in org.hibernate.engine.spi.ActionQueue executeActions()
11. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
12. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
13. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
14. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
15. AbstractEntityManagerImpl.java line 996
    in org.hibernate.ejb.AbstractEntityManagerImpl flush()
16. CmfEntityManager.java line 998
    in com.cloudera.cmf.persist.CmfEntityManager flush()
17. HardwareController.java line 514
    in com.cloudera.server.web.cmf.HardwareController deleteHosts()
18. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
19. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
20. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
21. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
22. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
23. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
24. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
25. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
26. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
27. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
28. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
29. Method.java line 606
    in java.lang.reflect.Method invoke()
30. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
31. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
32. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
33. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
34. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
35. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
36. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
37. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
38. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
39. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
40. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
41. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
42. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
43. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
44. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
45. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
46. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
47. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
48. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
50. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
52. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
56. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
58. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
60. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
61. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
62. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
63. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
64. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
65. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
66. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
67. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
68. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
69. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
70. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
71. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
72. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
73. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
74. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
75. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
76. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
77. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
78. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
79. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
80. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
81. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
82. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
83. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
84. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
85. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
86. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
87. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
88. Server.java line 326
    in org.mortbay.jetty.Server handle()
89. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
90. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
91. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
92. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
93. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
94. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
95. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

[#2]

I ran df -h, it shows:

Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.9G 5.3G 2.3G 70% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda8 29G 296M 27G 2% /apps
/dev/sda1 243M 32M 198M 14% /boot
/dev/sda5 7.9G 371M 7.2G 5% /home
/dev/sda7 4.0G 137M 3.7G 4% /tmp
/dev/sda6 7.9G 7.5G 0 100% /var
cm_processes 3.9G 133M 3.8G 4% /var/run/cloudera-scm-agent/process
/dev/sdm1 20G 172M 19G 1% /home/var

The /var is full. Is it possible that this has something to do with my problem? The error seems like it is saying that I'm not able to make insertion into the DB.

[#3]

Please stop entire cluster and run du to figure out where the space is
being used.

# du -ks /var/* | sort -n

Once you find the culprit, let us know and we can decide how to fix it as
the solution depends on the service.

[#4]

When you suggest stopping entire cluster, do you mean stopping the
cloudera-scm-server and all cloudera-scm-server-agent services on every
host?

Or is there a direct way to do this via Cloudera Manager? I'm sorry, I'm a
newbie at this.

Thanks for your reply.

[#5]

Visit Cloudera Manager, in the drop down list next to the cluster name
you'll see "stop cluster". Run that all services will stop cleanly. Then
check disk usage as suggested earlier.

[#6]

I tried to stop the cluster as you suggested via Cloudera Manager, I got the following error:

Path: http://ipaddress:port/cmf/clusters/1/do

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

javax.persistence.RollbackException:Error while committing the transaction
at TransactionImpl.java line 92
in org.hibernate.ejb.TransactionImpl commit()

 Stack Trace:

1. TransactionImpl.java line 92
   in org.hibernate.ejb.TransactionImpl commit()
2. AbstractWrappedEntityManager.java line 110
   in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
3. CmfEntityManager.java line 368
   in com.cloudera.cmf.persist.CmfEntityManager commit()
4. ClusterController.java line 195
   in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
5. <generated> line -1
   in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
6. MethodProxy.java line 191
   in net.sf.cglib.proxy.MethodProxy invoke()
7. Cglib2AopProxy.java line 688
   in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
8. ReflectiveMethodInvocation.java line 150
   in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
9. MethodSecurityInterceptor.java line 61
   in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
10. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
11. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
12. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
13. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
14. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
15. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
16. Method.java line 606
    in java.lang.reflect.Method invoke()
17. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
18. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
19. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
20. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
21. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
22. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
23. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
24. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
25. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
26. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
27. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
28. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
29. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
30. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
31. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
32. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
33. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
34. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
35. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
36. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
37. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
38. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
39. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
40. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
41. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
42. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
44. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
46. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
54. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
55. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
60. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
61. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
62. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
63. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
64. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
65. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
66. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
67. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
68. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
69. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
70. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
71. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
72. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
73. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
74. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
75. Server.java line 326
    in org.mortbay.jetty.Server handle()
76. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
77. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
78. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
79. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
80. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
81. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
82. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: could not perform addBatch
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()

 Stack Trace:

1. AbstractEntityManagerImpl.java line 1387
   in org.hibernate.ejb.AbstractEntityManagerImpl convert()
2. AbstractEntityManagerImpl.java line 1310
   in org.hibernate.ejb.AbstractEntityManagerImpl convert()
3. TransactionImpl.java line 80
   in org.hibernate.ejb.TransactionImpl commit()
4. AbstractWrappedEntityManager.java line 110
   in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
5. CmfEntityManager.java line 368
   in com.cloudera.cmf.persist.CmfEntityManager commit()
6. ClusterController.java line 195
   in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
7. <generated> line -1
   in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
8. MethodProxy.java line 191
   in net.sf.cglib.proxy.MethodProxy invoke()
9. Cglib2AopProxy.java line 688
   in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
10. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
11. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
12. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
13. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
14. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
15. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
16. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
17. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
18. Method.java line 606
    in java.lang.reflect.Method invoke()
19. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
20. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
21. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
22. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
23. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
24. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
25. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
26. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
27. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
28. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
29. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
30. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
31. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
32. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
33. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
34. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
35. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
36. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
37. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
38. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
39. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
40. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
41. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
42. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
43. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
44. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
45. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
46. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
47. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
49. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
51. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
53. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
54. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
55. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
60. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
61. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
62. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
63. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
64. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
65. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
66. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
67. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
68. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
69. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
70. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
71. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
72. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
73. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
74. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
75. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
76. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
77. Server.java line 326
    in org.mortbay.jetty.Server handle()
78. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
79. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
80. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
81. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
82. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
83. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
84. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: org.hibernate.exception.GenericJDBCException:could not perform addBatch
at StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()

 Stack Trace:

1. StandardSQLExceptionConverter.java line 54
   in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
2. SqlExceptionHelper.java line 125
   in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
3. BatchingBatch.java line 114
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
4. BatchingBatch.java line 101
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
5. AbstractBatchImpl.java line 149
   in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
6. JdbcCoordinatorImpl.java line 187
   in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl getBatch()
7. AbstractEntityPersister.java line 3045
   in org.hibernate.persister.entity.AbstractEntityPersister insert()
8. AbstractEntityPersister.java line 3509
   in org.hibernate.persister.entity.AbstractEntityPersister insert()
9. EntityInsertAction.java line 88
   in org.hibernate.action.internal.EntityInsertAction execute()
10. ActionQueue.java line 377
    in org.hibernate.engine.spi.ActionQueue execute()
11. ActionQueue.java line 369
    in org.hibernate.engine.spi.ActionQueue executeActions()
12. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
13. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
14. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
15. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
16. SessionImpl.java line 404
    in org.hibernate.internal.SessionImpl managedFlush()
17. JdbcTransaction.java line 101
    in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction beforeTransactionCommit()
18. AbstractTransactionImpl.java line 175
    in org.hibernate.engine.transaction.spi.AbstractTransactionImpl commit()
19. TransactionImpl.java line 75
    in org.hibernate.ejb.TransactionImpl commit()
20. AbstractWrappedEntityManager.java line 110
    in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
21. CmfEntityManager.java line 368
    in com.cloudera.cmf.persist.CmfEntityManager commit()
22. ClusterController.java line 195
    in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
23. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
24. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
25. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
26. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
27. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
28. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
29. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
30. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
31. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
32. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
33. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
34. Method.java line 606
    in java.lang.reflect.Method invoke()
35. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
36. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
37. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
38. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
39. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
40. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
41. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
42. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
43. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
44. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
45. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
46. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
47. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
48. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
49. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
50. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
51. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
52. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
53. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
54. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
55. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
57. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
59. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
60. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
61. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
62. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
63. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
64. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
65. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
66. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
67. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
68. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
69. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
70. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
71. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
72. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
73. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
74. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
75. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
76. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
77. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
78. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
79. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
80. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
81. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
82. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
83. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
84. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
85. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
86. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
87. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
88. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
89. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
90. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
91. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
92. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
93. Server.java line 326
    in org.mortbay.jetty.Server handle()
94. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
95. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
96. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
97. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
98. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
99. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
100. QueuedThreadPool.java line 582
     in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.sql.BatchUpdateException:Batch entry 0 insert into COMMANDS (OPTIMISTIC_LOCK_VERSION, NAME, STATE, START_INSTANT, END_INSTANT, ACTIVE, RESULT_MESSAGE, RESULT_DATA, RESULT_DATA_MIME_TYPE, RESULT_DATA_FILENAME, RESULT_DATA_PATH, RESULT_DATA_REAPED, SUCCESS, INTERNAL_STATE, ARGUMENTS, AUDITED, CLUSTER_ID, SERVICE_ID, ROLE_ID, HOST_ID, SCHEDULE_ID, PARENT_ID, COMMAND_ID) values ('0', 'Stop', 'STARTED', '1409659192831', NULL, '1', NULL, NULL, NULL, NULL, NULL, '0', '0', NULL, NULL, '0', '1', NULL, NULL, NULL, NULL, NULL, '256') was aborted. Call getNextException to see the cause.
at AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()

 Stack Trace:

1. AbstractJdbc2Statement.java line 2598
   in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
2. QueryExecutorImpl.java line 1836
   in org.postgresql.core.v3.QueryExecutorImpl processResults()
3. QueryExecutorImpl.java line 407
   in org.postgresql.core.v3.QueryExecutorImpl execute()
4. AbstractJdbc2Statement.java line 2737
   in org.postgresql.jdbc2.AbstractJdbc2Statement executeBatch()
5. NewProxyPreparedStatement.java line 1723
   in com.mchange.v2.c3p0.impl.NewProxyPreparedStatement executeBatch()
6. BatchingBatch.java line 110
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
7. BatchingBatch.java line 101
   in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
8. AbstractBatchImpl.java line 149
   in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
9. JdbcCoordinatorImpl.java line 187
   in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl getBatch()
10. AbstractEntityPersister.java line 3045
    in org.hibernate.persister.entity.AbstractEntityPersister insert()
11. AbstractEntityPersister.java line 3509
    in org.hibernate.persister.entity.AbstractEntityPersister insert()
12. EntityInsertAction.java line 88
    in org.hibernate.action.internal.EntityInsertAction execute()
13. ActionQueue.java line 377
    in org.hibernate.engine.spi.ActionQueue execute()
14. ActionQueue.java line 369
    in org.hibernate.engine.spi.ActionQueue executeActions()
15. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
16. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
17. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
18. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
19. SessionImpl.java line 404
    in org.hibernate.internal.SessionImpl managedFlush()
20. JdbcTransaction.java line 101
    in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction beforeTransactionCommit()
21. AbstractTransactionImpl.java line 175
    in org.hibernate.engine.transaction.spi.AbstractTransactionImpl commit()
22. TransactionImpl.java line 75
    in org.hibernate.ejb.TransactionImpl commit()
23. AbstractWrappedEntityManager.java line 110
    in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
24. CmfEntityManager.java line 368
    in com.cloudera.cmf.persist.CmfEntityManager 

[#7]

The database is unable to write so shutdown is failing. Please run the du
command from earlier and see which directory is using the most space

[#8]

Here's what I got:

0 /var/mail
4 /var/account
4 /var/crash
4 /var/cvs
4 /var/games
4 /var/local
4 /var/nis
4 /var/opt
4 /var/preserve
4 /var/tmp
4 /var/yp
8 /var/empty
16 /var/lock
16 /var/lost+found
28 /var/db
140 /var/spool
125084 /var/cache
136760 /var/run
347180 /var/log
7221464 /var/lib

Thank you for the reply.

[#9]

Repeat the exercise within /var/lib. Most likely it is the event server.

[#10]

Here's the result largest to smallest:

5074352 /var/lib/cloudera-service-monitor
1888312 /var/lib/cloudera-host-monitor
97444 /var/lib/cloudera-scm-server-db
47992 /var/lib/rpm
33988 /var/lib/solr
25312 /var/lib/cloudera-scm-eventserver
17720 /var/lib/zookeeper
11608 /var/lib/sqoop2
9880 /var/lib/yum
4556 /var/lib/cloudera-scm-server
4276 /var/lib/oozie
3928 /var/lib/mlocate
804 /var/lib/cloudera-scm-headlamp
240 /var/lib/hue
132 /var/lib/cloudera-scm-agent
132 /var/lib/alternatives
36 /var/lib/readahead
32 /var/lib/cas
28 /var/lib/polkit-1
24 /var/lib/nfs
20 /var/lib/rhsm
16 /var/lib/samba
16 /var/lib/pgsql
12 /var/lib/stateless
12 /var/lib/prelink
8 /var/lib/udev
8 /var/lib/postfix
8 /var/lib/plymouth
8 /var/lib/hive
8 /var/lib/hadoop-yarn
8 /var/lib/hadoop-mapreduce
8 /var/lib/hadoop-hdfs
8 /var/lib/dbus
8 /var/lib/authconfig
4 /var/lib/up2date
4 /var/lib/sqoop
4 /var/lib/spark
4 /var/lib/rsyslog
4 /var/lib/random-seed
4 /var/lib/ntp
4 /var/lib/misc
4 /var/lib/logrotate.status
4 /var/lib/impala
4 /var/lib/hive-hcatalog
4 /var/lib/hadoop-httpfs
4 /var/lib/games
4 /var/lib/fprint
4 /var/lib/flume-ng
4 /var/lib/dhclient
4 /var/lib/bigtop

I forgot to mention that before I ran into thi error, symlink in /home/var has been created to extend /var using extended disk.

Thanks for the reply.