Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Unable to delete hosts or assign roles

avatar
Explorer

Hi,

 

I'm currently running Cloudera Manager 5.0.2. I am having problem when I try to delete host. And also when I try to assign roles to host.

 

I added 2 hosts into cluster:

 

1. Host 1 did not pass host inspection because I installed the wrong CDH version (5.1.2); other hosts are running CDH 5.0.3.

2. Host 2 installed successfully, but I run into error when I try to add role.

 

The error message is different, but I feel that the 2 problems are related somehow.

 

Here's the error log that I received when I try to delete host:

 

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: could not perform addBatch
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()
 Stack Trace:
Caused by: org.hibernate.exception.GenericJDBCException:could not perform addBatch
at StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
 Stack Trace:
  1. StandardSQLExceptionConverter.java line 54
    in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
  2. SqlExceptionHelper.java line 125
    in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
  3. BatchingBatch.java line 114
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
  4. BatchingBatch.java line 101
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
  5. AbstractBatchImpl.java line 149
    in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
  6. JdbcCoordinatorImpl.java line 198
    in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
  7. ActionQueue.java line 372
    in org.hibernate.engine.spi.ActionQueue executeActions()
  8. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
  9. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
  10. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
  11. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
  12. AbstractEntityManagerImpl.java line 996
    in org.hibernate.ejb.AbstractEntityManagerImpl flush()
  13. CmfEntityManager.java line 998
    in com.cloudera.cmf.persist.CmfEntityManager flush()
  14. HardwareController.java line 514
    in com.cloudera.server.web.cmf.HardwareController deleteHosts()
  15. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
  16. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
  17. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
  18. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  19. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
  20. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  21. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
  22. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
  23. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
  24. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
  25. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
  26. Method.java line 606
    in java.lang.reflect.Method invoke()
  27. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
  28. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
  29. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
  30. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
  31. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
  32. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
  33. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
  34. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
  35. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
  36. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
  37. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  38. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
  39. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
  40. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  41. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
  42. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  43. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
  44. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  45. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  46. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
  47. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
  48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  49. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
  50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  51. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
  52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  53. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
  54. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  55. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
  56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  57. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
  58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  59. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
  60. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  61. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
  62. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  63. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
  64. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  65. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
  66. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  67. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
  68. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  69. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
  70. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
  71. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
  72. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  73. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
  74. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
  75. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  76. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
  77. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  78. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
  79. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  80. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
  81. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
  82. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  83. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
  84. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  85. Server.java line 326
    in org.mortbay.jetty.Server handle()
  86. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
  87. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
  88. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
  89. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
  90. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
  91. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
  92. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()
Caused by: java.sql.BatchUpdateException:Batch entry 0 insert into AUDITS (OPTIMISTIC_LOCK_VERSION, ACTING_USER_ID, AUDIT_TYPE, CLUSTER_ID, SERVICE_ID, ROLE_ID, COMMAND_ID, USER_ID, HOST_ID, HOST_TEMPLATE_ID, CONFIG_CONTAINER_ID, CREATED_INSTANT, MESSAGE, IP_ADDRESS, AUDIT_ID) values ('0', '1', 'HOST', NULL, NULL, NULL, NULL, NULL, '4', NULL, NULL, '1409644295347', 'Deleted host analyticpapp6.telkomsel.co.id', '10.2.133.134', '203') was aborted. Call getNextException to see the cause.
at AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
 Stack Trace:
  1. AbstractJdbc2Statement.java line 2598
    in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
  2. QueryExecutorImpl.java line 1836
    in org.postgresql.core.v3.QueryExecutorImpl processResults()
  3. QueryExecutorImpl.java line 407
    in org.postgresql.core.v3.QueryExecutorImpl execute()
  4. AbstractJdbc2Statement.java line 2737
    in org.postgresql.jdbc2.AbstractJdbc2Statement executeBatch()
  5. NewProxyPreparedStatement.java line 1723
    in com.mchange.v2.c3p0.impl.NewProxyPreparedStatement executeBatch()
  6. BatchingBatch.java line 110
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
  7. BatchingBatch.java line 101
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
  8. AbstractBatchImpl.java line 149
    in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
  9. JdbcCoordinatorImpl.java line 198
    in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
  10. ActionQueue.java line 372
    in org.hibernate.engine.spi.ActionQueue executeActions()
  11. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
  12. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
  13. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
  14. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
  15. AbstractEntityManagerImpl.java line 996
    in org.hibernate.ejb.AbstractEntityManagerImpl flush()
  16. CmfEntityManager.java line 998
    in com.cloudera.cmf.persist.CmfEntityManager flush()
  17. HardwareController.java line 514
    in com.cloudera.server.web.cmf.HardwareController deleteHosts()
  18. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
  19. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
  20. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
  21. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  22. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
  23. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  24. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
  25. <generated> line -1
    in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
  26. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
  27. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
  28. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
  29. Method.java line 606
    in java.lang.reflect.Method invoke()
  30. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
  31. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
  32. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
  33. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
  34. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
  35. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
  36. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
  37. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
  38. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
  39. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
  40. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  41. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
  42. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
  43. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  44. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
  45. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  46. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
  47. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  48. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  49. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
  50. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
  51. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  52. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
  53. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  54. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
  55. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  56. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
  57. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  58. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
  59. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  60. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
  61. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  62. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
  63. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  64. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
  65. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  66. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
  67. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  68. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
  69. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  70. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
  71. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  72. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
  73. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
  74. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
  75. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  76. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
  77. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
  78. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  79. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
  80. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  81. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
  82. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  83. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
  84. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
  85. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  86. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
  87. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  88. Server.java line 326
    in org.mortbay.jetty.Server handle()
  89. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
  90. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
  91. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
  92. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
  93. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
  94. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
  95. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()
1 ACCEPTED SOLUTION

avatar
Explorer

I've solved this issue.

 

Cloudera Embedded DB cannot write because it is located in my /var directory (under /var/lib/cloudera-scm-server-db) which is full. I use this step-by-step solution (credits to puneethabm from google group):

 

1) Stop the scm service:
# service cloudera-scm-server stop
# service cloudera-scm-server-db stop 
 
2) Copy the cloudera-scm-server-db, retaining the permissions as below:
#cd /var/lib
#cp -rp cloudera-scm-server-db /dir1/lib/
 
#cd /var/lib/cloudera-scm-server-db
#rm -rf data/
 
3) Create symlink
#cd /var/lib/cloudera-scm-server-db
#ln -s /dir1/lib/cloudera-scm-server-db/data data 
 
4) Start services:
# service cloudera-scm-server-db start
DB initialization done.
waiting for server to start.... done
server started
# service cloudera-scm-server start
 
By creating symlink to disk with sufficient free space, it allows Cloudera Embedded DB to perform write transaction again.

View solution in original post

12 REPLIES 12

avatar
Explorer
I ran df -h, it shows:

Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.9G 5.3G 2.3G 70% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda8 29G 296M 27G 2% /apps
/dev/sda1 243M 32M 198M 14% /boot
/dev/sda5 7.9G 371M 7.2G 5% /home
/dev/sda7 4.0G 137M 3.7G 4% /tmp
/dev/sda6 7.9G 7.5G 0 100% /var
cm_processes 3.9G 133M 3.8G 4% /var/run/cloudera-scm-agent/process
/dev/sdm1 20G 172M 19G 1% /home/var

The /var is full. Is it possible that this has something to do with my problem? The error seems like it is saying that I'm not able to make insertion into the DB.

avatar
Please stop entire cluster and run du to figure out where the space is
being used.

# du -ks /var/* | sort -n

Once you find the culprit, let us know and we can decide how to fix it as
the solution depends on the service.

Regards,
Gautam Gopalakrishnan

avatar
Explorer
When you suggest stopping entire cluster, do you mean stopping the
cloudera-scm-server and all cloudera-scm-server-agent services on every
host?

Or is there a direct way to do this via Cloudera Manager? I'm sorry, I'm a
newbie at this.

Thanks for your reply.

avatar
Visit Cloudera Manager, in the drop down list next to the cluster name
you'll see "stop cluster". Run that all services will stop cleanly. Then
check disk usage as suggested earlier.

Regards,
Gautam Gopalakrishnan

avatar
Explorer

I tried to stop the cluster as you suggested via Cloudera Manager, I got the following error:

 

Path: http://ipaddress:port/cmf/clusters/1/do

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

javax.persistence.RollbackException:Error while committing the transaction
at TransactionImpl.java line 92
in org.hibernate.ejb.TransactionImpl commit()
 Stack Trace:
  1. TransactionImpl.java line 92
    in org.hibernate.ejb.TransactionImpl commit()
  2. AbstractWrappedEntityManager.java line 110
    in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
  3. CmfEntityManager.java line 368
    in com.cloudera.cmf.persist.CmfEntityManager commit()
  4. ClusterController.java line 195
    in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
  5. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
  6. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
  7. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
  8. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  9. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
  10. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  11. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
  12. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
  13. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
  14. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
  15. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
  16. Method.java line 606
    in java.lang.reflect.Method invoke()
  17. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
  18. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
  19. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
  20. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
  21. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
  22. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
  23. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
  24. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
  25. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
  26. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
  27. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  28. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
  29. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
  30. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  31. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
  32. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  33. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
  34. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  35. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  36. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
  37. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
  38. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  39. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
  40. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  41. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
  42. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  43. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
  44. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  45. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
  46. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  47. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
  48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  49. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
  50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  51. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
  52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  53. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
  54. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  55. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
  56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  57. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
  58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  59. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
  60. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
  61. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
  62. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  63. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
  64. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
  65. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  66. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
  67. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  68. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
  69. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  70. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
  71. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
  72. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  73. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
  74. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  75. Server.java line 326
    in org.mortbay.jetty.Server handle()
  76. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
  77. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
  78. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
  79. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
  80. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
  81. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
  82. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()
Caused by: javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: could not perform addBatch
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()
 Stack Trace:
  1. AbstractEntityManagerImpl.java line 1387
    in org.hibernate.ejb.AbstractEntityManagerImpl convert()
  2. AbstractEntityManagerImpl.java line 1310
    in org.hibernate.ejb.AbstractEntityManagerImpl convert()
  3. TransactionImpl.java line 80
    in org.hibernate.ejb.TransactionImpl commit()
  4. AbstractWrappedEntityManager.java line 110
    in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
  5. CmfEntityManager.java line 368
    in com.cloudera.cmf.persist.CmfEntityManager commit()
  6. ClusterController.java line 195
    in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
  7. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
  8. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
  9. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
  10. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  11. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
  12. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  13. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
  14. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
  15. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
  16. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
  17. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
  18. Method.java line 606
    in java.lang.reflect.Method invoke()
  19. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
  20. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
  21. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
  22. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
  23. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
  24. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
  25. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
  26. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
  27. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
  28. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
  29. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  30. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
  31. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
  32. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  33. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
  34. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  35. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
  36. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  37. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  38. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
  39. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
  40. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  41. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
  42. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  43. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
  44. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  45. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
  46. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  47. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
  48. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  49. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
  50. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  51. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
  52. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  53. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
  54. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  55. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
  56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  57. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
  58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  59. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
  60. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  61. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
  62. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
  63. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
  64. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  65. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
  66. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
  67. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  68. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
  69. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  70. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
  71. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  72. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
  73. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
  74. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  75. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
  76. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  77. Server.java line 326
    in org.mortbay.jetty.Server handle()
  78. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
  79. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
  80. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
  81. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
  82. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
  83. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
  84. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()
Caused by: org.hibernate.exception.GenericJDBCException:could not perform addBatch
at StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
 Stack Trace:
  1. StandardSQLExceptionConverter.java line 54
    in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
  2. SqlExceptionHelper.java line 125
    in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
  3. BatchingBatch.java line 114
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
  4. BatchingBatch.java line 101
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
  5. AbstractBatchImpl.java line 149
    in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
  6. JdbcCoordinatorImpl.java line 187
    in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl getBatch()
  7. AbstractEntityPersister.java line 3045
    in org.hibernate.persister.entity.AbstractEntityPersister insert()
  8. AbstractEntityPersister.java line 3509
    in org.hibernate.persister.entity.AbstractEntityPersister insert()
  9. EntityInsertAction.java line 88
    in org.hibernate.action.internal.EntityInsertAction execute()
  10. ActionQueue.java line 377
    in org.hibernate.engine.spi.ActionQueue execute()
  11. ActionQueue.java line 369
    in org.hibernate.engine.spi.ActionQueue executeActions()
  12. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
  13. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
  14. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
  15. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
  16. SessionImpl.java line 404
    in org.hibernate.internal.SessionImpl managedFlush()
  17. JdbcTransaction.java line 101
    in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction beforeTransactionCommit()
  18. AbstractTransactionImpl.java line 175
    in org.hibernate.engine.transaction.spi.AbstractTransactionImpl commit()
  19. TransactionImpl.java line 75
    in org.hibernate.ejb.TransactionImpl commit()
  20. AbstractWrappedEntityManager.java line 110
    in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
  21. CmfEntityManager.java line 368
    in com.cloudera.cmf.persist.CmfEntityManager commit()
  22. ClusterController.java line 195
    in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
  23. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
  24. MethodProxy.java line 191
    in net.sf.cglib.proxy.MethodProxy invoke()
  25. Cglib2AopProxy.java line 688
    in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
  26. ReflectiveMethodInvocation.java line 150
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  27. MethodSecurityInterceptor.java line 61
    in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
  28. ReflectiveMethodInvocation.java line 172
    in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
  29. Cglib2AopProxy.java line 621
    in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
  30. <generated> line -1
    in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
  31. NativeMethodAccessorImpl.java line -2
    in sun.reflect.NativeMethodAccessorImpl invoke0()
  32. NativeMethodAccessorImpl.java line 57
    in sun.reflect.NativeMethodAccessorImpl invoke()
  33. DelegatingMethodAccessorImpl.java line 43
    in sun.reflect.DelegatingMethodAccessorImpl invoke()
  34. Method.java line 606
    in java.lang.reflect.Method invoke()
  35. HandlerMethodInvoker.java line 176
    in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
  36. AnnotationMethodHandlerAdapter.java line 436
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
  37. AnnotationMethodHandlerAdapter.java line 424
    in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
  38. DispatcherServlet.java line 790
    in org.springframework.web.servlet.DispatcherServlet doDispatch()
  39. DispatcherServlet.java line 719
    in org.springframework.web.servlet.DispatcherServlet doService()
  40. FrameworkServlet.java line 669
    in org.springframework.web.servlet.FrameworkServlet processRequest()
  41. FrameworkServlet.java line 585
    in org.springframework.web.servlet.FrameworkServlet doPost()
  42. HttpServlet.java line 727
    in javax.servlet.http.HttpServlet service()
  43. HttpServlet.java line 820
    in javax.servlet.http.HttpServlet service()
  44. ServletHolder.java line 511
    in org.mortbay.jetty.servlet.ServletHolder handle()
  45. ServletHandler.java line 1221
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  46. UserAgentFilter.java line 78
    in org.mortbay.servlet.UserAgentFilter doFilter()
  47. GzipFilter.java line 131
    in org.mortbay.servlet.GzipFilter doFilter()
  48. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  49. JAMonServletFilter.java line 48
    in com.jamonapi.http.JAMonServletFilter doFilter()
  50. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  51. JavaMelodyFacade.java line 109
    in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
  52. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  53. FilterChainProxy.java line 311
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  54. FilterSecurityInterceptor.java line 116
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
  55. FilterSecurityInterceptor.java line 83
    in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
  56. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  57. ExceptionTranslationFilter.java line 113
    in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
  58. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  59. SessionManagementFilter.java line 101
    in org.springframework.security.web.session.SessionManagementFilter doFilter()
  60. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  61. AnonymousAuthenticationFilter.java line 113
    in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
  62. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  63. RememberMeAuthenticationFilter.java line 146
    in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
  64. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  65. SecurityContextHolderAwareRequestFilter.java line 54
    in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
  66. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  67. RequestCacheAwareFilter.java line 45
    in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
  68. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  69. AbstractAuthenticationProcessingFilter.java line 182
    in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
  70. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  71. LogoutFilter.java line 105
    in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
  72. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  73. SecurityContextPersistenceFilter.java line 87
    in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
  74. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  75. ConcurrentSessionFilter.java line 125
    in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
  76. FilterChainProxy.java line 323
    in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
  77. FilterChainProxy.java line 173
    in org.springframework.security.web.FilterChainProxy doFilter()
  78. DelegatingFilterProxy.java line 237
    in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
  79. DelegatingFilterProxy.java line 167
    in org.springframework.web.filter.DelegatingFilterProxy doFilter()
  80. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  81. CharacterEncodingFilter.java line 88
    in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
  82. OncePerRequestFilter.java line 76
    in org.springframework.web.filter.OncePerRequestFilter doFilter()
  83. ServletHandler.java line 1212
    in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
  84. ServletHandler.java line 399
    in org.mortbay.jetty.servlet.ServletHandler handle()
  85. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  86. SessionHandler.java line 182
    in org.mortbay.jetty.servlet.SessionHandler handle()
  87. SecurityHandler.java line 216
    in org.mortbay.jetty.security.SecurityHandler handle()
  88. ContextHandler.java line 766
    in org.mortbay.jetty.handler.ContextHandler handle()
  89. WebAppContext.java line 450
    in org.mortbay.jetty.webapp.WebAppContext handle()
  90. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  91. StatisticsHandler.java line 53
    in org.mortbay.jetty.handler.StatisticsHandler handle()
  92. HandlerWrapper.java line 152
    in org.mortbay.jetty.handler.HandlerWrapper handle()
  93. Server.java line 326
    in org.mortbay.jetty.Server handle()
  94. HttpConnection.java line 542
    in org.mortbay.jetty.HttpConnection handleRequest()
  95. HttpConnection.java line 945
    in org.mortbay.jetty.HttpConnection$RequestHandler content()
  96. HttpParser.java line 756
    in org.mortbay.jetty.HttpParser parseNext()
  97. HttpParser.java line 218
    in org.mortbay.jetty.HttpParser parseAvailable()
  98. HttpConnection.java line 404
    in org.mortbay.jetty.HttpConnection handle()
  99. SelectChannelEndPoint.java line 410
    in org.mortbay.io.nio.SelectChannelEndPoint run()
  100. QueuedThreadPool.java line 582
    in org.mortbay.thread.QueuedThreadPool$PoolThread run()
Caused by: java.sql.BatchUpdateException:Batch entry 0 insert into COMMANDS (OPTIMISTIC_LOCK_VERSION, NAME, STATE, START_INSTANT, END_INSTANT, ACTIVE, RESULT_MESSAGE, RESULT_DATA, RESULT_DATA_MIME_TYPE, RESULT_DATA_FILENAME, RESULT_DATA_PATH, RESULT_DATA_REAPED, SUCCESS, INTERNAL_STATE, ARGUMENTS, AUDITED, CLUSTER_ID, SERVICE_ID, ROLE_ID, HOST_ID, SCHEDULE_ID, PARENT_ID, COMMAND_ID) values ('0', 'Stop', 'STARTED', '1409659192831', NULL, '1', NULL, NULL, NULL, NULL, NULL, '0', '0', NULL, NULL, '0', '1', NULL, NULL, NULL, NULL, NULL, '256') was aborted. Call getNextException to see the cause.
at AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
 Stack Trace:
  1. AbstractJdbc2Statement.java line 2598
    in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
  2. QueryExecutorImpl.java line 1836
    in org.postgresql.core.v3.QueryExecutorImpl processResults()
  3. QueryExecutorImpl.java line 407
    in org.postgresql.core.v3.QueryExecutorImpl execute()
  4. AbstractJdbc2Statement.java line 2737
    in org.postgresql.jdbc2.AbstractJdbc2Statement executeBatch()
  5. NewProxyPreparedStatement.java line 1723
    in com.mchange.v2.c3p0.impl.NewProxyPreparedStatement executeBatch()
  6. BatchingBatch.java line 110
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
  7. BatchingBatch.java line 101
    in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
  8. AbstractBatchImpl.java line 149
    in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
  9. JdbcCoordinatorImpl.java line 187
    in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl getBatch()
  10. AbstractEntityPersister.java line 3045
    in org.hibernate.persister.entity.AbstractEntityPersister insert()
  11. AbstractEntityPersister.java line 3509
    in org.hibernate.persister.entity.AbstractEntityPersister insert()
  12. EntityInsertAction.java line 88
    in org.hibernate.action.internal.EntityInsertAction execute()
  13. ActionQueue.java line 377
    in org.hibernate.engine.spi.ActionQueue execute()
  14. ActionQueue.java line 369
    in org.hibernate.engine.spi.ActionQueue executeActions()
  15. ActionQueue.java line 286
    in org.hibernate.engine.spi.ActionQueue executeActions()
  16. AbstractFlushingEventListener.java line 339
    in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
  17. DefaultFlushEventListener.java line 52
    in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
  18. SessionImpl.java line 1234
    in org.hibernate.internal.SessionImpl flush()
  19. SessionImpl.java line 404
    in org.hibernate.internal.SessionImpl managedFlush()
  20. JdbcTransaction.java line 101
    in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction beforeTransactionCommit()
  21. AbstractTransactionImpl.java line 175
    in org.hibernate.engine.transaction.spi.AbstractTransactionImpl commit()
  22. TransactionImpl.java line 75
    in org.hibernate.ejb.TransactionImpl commit()
  23. AbstractWrappedEntityManager.java line 110
    in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
  24. CmfEntityManager.java line 368
    in com.cloudera.cmf.persist.CmfEntityManager 

avatar
The database is unable to write so shutdown is failing. Please run the du
command from earlier and see which directory is using the most space

Regards,
Gautam Gopalakrishnan

avatar
Explorer

Here's what I got:

 

0 /var/mail
4 /var/account
4 /var/crash
4 /var/cvs
4 /var/games
4 /var/local
4 /var/nis
4 /var/opt
4 /var/preserve
4 /var/tmp
4 /var/yp
8 /var/empty
16 /var/lock
16 /var/lost+found
28 /var/db
140 /var/spool
125084 /var/cache
136760 /var/run
347180 /var/log
7221464 /var/lib

 

Thank you for the reply.

avatar
Repeat the exercise within /var/lib. Most likely it is the event server.

Regards,
Gautam Gopalakrishnan

avatar
Explorer

Here's the result largest to smallest:

 

5074352 /var/lib/cloudera-service-monitor
1888312 /var/lib/cloudera-host-monitor
97444 /var/lib/cloudera-scm-server-db
47992 /var/lib/rpm
33988 /var/lib/solr
25312 /var/lib/cloudera-scm-eventserver
17720 /var/lib/zookeeper
11608 /var/lib/sqoop2
9880 /var/lib/yum
4556 /var/lib/cloudera-scm-server
4276 /var/lib/oozie
3928 /var/lib/mlocate
804 /var/lib/cloudera-scm-headlamp
240 /var/lib/hue
132 /var/lib/cloudera-scm-agent
132 /var/lib/alternatives
36 /var/lib/readahead
32 /var/lib/cas
28 /var/lib/polkit-1
24 /var/lib/nfs
20 /var/lib/rhsm
16 /var/lib/samba
16 /var/lib/pgsql
12 /var/lib/stateless
12 /var/lib/prelink
8 /var/lib/udev
8 /var/lib/postfix
8 /var/lib/plymouth
8 /var/lib/hive
8 /var/lib/hadoop-yarn
8 /var/lib/hadoop-mapreduce
8 /var/lib/hadoop-hdfs
8 /var/lib/dbus
8 /var/lib/authconfig
4 /var/lib/up2date
4 /var/lib/sqoop
4 /var/lib/spark
4 /var/lib/rsyslog
4 /var/lib/random-seed
4 /var/lib/ntp
4 /var/lib/misc
4 /var/lib/logrotate.status
4 /var/lib/impala
4 /var/lib/hive-hcatalog
4 /var/lib/hadoop-httpfs
4 /var/lib/games
4 /var/lib/fprint
4 /var/lib/flume-ng
4 /var/lib/dhclient
4 /var/lib/bigtop

 

I forgot to mention that before I ran into thi error, symlink in /home/var has been created to extend /var using extended disk.

 

Thanks for the reply.