question Re: Unable to delete hosts or assign roles in Archives of Support Questions (Read Only)

Unable to delete hosts or assign roles

gagabonar — Fri, 16 Sep 2022 09:06:33 GMT

Hi,

I'm currently running Cloudera Manager 5.0.2. I am having problem when I try to delete host. And also when I try to assign roles to host.

I added 2 hosts into cluster:

1. Host 1 did not pass host inspection because I installed the wrong CDH version (5.1.2); other hosts are running CDH 5.0.3.

2. Host 2 installed successfully, but I run into error when I try to add role.

The error message is different, but I feel that the 2 problems are related somehow.

Here's the error log that I received when I try to delete host:

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: could not perform addBatch
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()

Stack Trace:

Caused by: org.hibernate.exception.GenericJDBCException:could not perform addBatch
at StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()

Stack Trace:

StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
SqlExceptionHelper.java line 125
in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
BatchingBatch.java line 114
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
BatchingBatch.java line 101
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
AbstractBatchImpl.java line 149
in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
JdbcCoordinatorImpl.java line 198
in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
ActionQueue.java line 372
in org.hibernate.engine.spi.ActionQueue executeActions()
ActionQueue.java line 286
in org.hibernate.engine.spi.ActionQueue executeActions()
AbstractFlushingEventListener.java line 339
in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
DefaultFlushEventListener.java line 52
in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
SessionImpl.java line 1234
in org.hibernate.internal.SessionImpl flush()
AbstractEntityManagerImpl.java line 996
in org.hibernate.ejb.AbstractEntityManagerImpl flush()
CmfEntityManager.java line 998
in com.cloudera.cmf.persist.CmfEntityManager flush()
HardwareController.java line 514
in com.cloudera.server.web.cmf.HardwareController deleteHosts()
<generated> line -1
in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
MethodProxy.java line 191
in net.sf.cglib.proxy.MethodProxy invoke()
Cglib2AopProxy.java line 688
in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
ReflectiveMethodInvocation.java line 150
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
MethodSecurityInterceptor.java line 61
in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
ReflectiveMethodInvocation.java line 172
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
Cglib2AopProxy.java line 621
in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
<generated> line -1
in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 57
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 606
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 585
in org.springframework.web.servlet.FrameworkServlet doPost()
HttpServlet.java line 727
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 820
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 766
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 945
in org.mortbay.jetty.HttpConnection$RequestHandler content()
HttpParser.java line 756
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 218
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.sql.BatchUpdateException:Batch entry 0 insert into AUDITS (OPTIMISTIC_LOCK_VERSION, ACTING_USER_ID, AUDIT_TYPE, CLUSTER_ID, SERVICE_ID, ROLE_ID, COMMAND_ID, USER_ID, HOST_ID, HOST_TEMPLATE_ID, CONFIG_CONTAINER_ID, CREATED_INSTANT, MESSAGE, IP_ADDRESS, AUDIT_ID) values ('0', '1', 'HOST', NULL, NULL, NULL, NULL, NULL, '4', NULL, NULL, '1409644295347', 'Deleted host analyticpapp6.telkomsel.co.id', '10.2.133.134', '203') was aborted. Call getNextException to see the cause.
at AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()

Stack Trace:

AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
QueryExecutorImpl.java line 1836
in org.postgresql.core.v3.QueryExecutorImpl processResults()
QueryExecutorImpl.java line 407
in org.postgresql.core.v3.QueryExecutorImpl execute()
AbstractJdbc2Statement.java line 2737
in org.postgresql.jdbc2.AbstractJdbc2Statement executeBatch()
NewProxyPreparedStatement.java line 1723
in com.mchange.v2.c3p0.impl.NewProxyPreparedStatement executeBatch()
BatchingBatch.java line 110
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
BatchingBatch.java line 101
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
AbstractBatchImpl.java line 149
in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
JdbcCoordinatorImpl.java line 198
in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl executeBatch()
ActionQueue.java line 372
in org.hibernate.engine.spi.ActionQueue executeActions()
ActionQueue.java line 286
in org.hibernate.engine.spi.ActionQueue executeActions()
AbstractFlushingEventListener.java line 339
in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
DefaultFlushEventListener.java line 52
in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
SessionImpl.java line 1234
in org.hibernate.internal.SessionImpl flush()
AbstractEntityManagerImpl.java line 996
in org.hibernate.ejb.AbstractEntityManagerImpl flush()
CmfEntityManager.java line 998
in com.cloudera.cmf.persist.CmfEntityManager flush()
HardwareController.java line 514
in com.cloudera.server.web.cmf.HardwareController deleteHosts()
<generated> line -1
in com.cloudera.server.web.cmf.HardwareController$$FastClassByCGLIB$$fcce8e79 invoke()
MethodProxy.java line 191
in net.sf.cglib.proxy.MethodProxy invoke()
Cglib2AopProxy.java line 688
in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
ReflectiveMethodInvocation.java line 150
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
MethodSecurityInterceptor.java line 61
in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
ReflectiveMethodInvocation.java line 172
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
Cglib2AopProxy.java line 621
in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
<generated> line -1
in com.cloudera.server.web.cmf.HardwareController$$EnhancerByCGLIB$$4396e333 deleteHosts()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 57
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 606
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 585
in org.springframework.web.servlet.FrameworkServlet doPost()
HttpServlet.java line 727
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 820
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 766
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 945
in org.mortbay.jetty.HttpConnection$RequestHandler content()
HttpParser.java line 756
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 218
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Re: Unable to delete hosts or assign roles

gagabonar — Tue, 02 Sep 2014 09:12:04 GMT

I ran df -h, it shows:

Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.9G 5.3G 2.3G 70% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda8 29G 296M 27G 2% /apps
/dev/sda1 243M 32M 198M 14% /boot
/dev/sda5 7.9G 371M 7.2G 5% /home
/dev/sda7 4.0G 137M 3.7G 4% /tmp
/dev/sda6 7.9G 7.5G 0 100% /var
cm_processes 3.9G 133M 3.8G 4% /var/run/cloudera-scm-agent/process
/dev/sdm1 20G 172M 19G 1% /home/var

The /var is full. Is it possible that this has something to do with my problem? The error seems like it is saying that I'm not able to make insertion into the DB.

Re: Unable to delete hosts or assign roles

GautamG — Tue, 02 Sep 2014 09:16:35 GMT

Please stop entire cluster and run du to figure out where the space is
being used.

# du -ks /var/* | sort -n

Once you find the culprit, let us know and we can decide how to fix it as
the solution depends on the service.

Re: Unable to delete hosts or assign roles

gagabonar — Tue, 02 Sep 2014 09:28:35 GMT

When you suggest stopping entire cluster, do you mean stopping the
cloudera-scm-server and all cloudera-scm-server-agent services on every
host?

Or is there a direct way to do this via Cloudera Manager? I'm sorry, I'm a
newbie at this.

Thanks for your reply.

Re: Unable to delete hosts or assign roles

GautamG — Tue, 02 Sep 2014 09:36:35 GMT

Visit Cloudera Manager, in the drop down list next to the cluster name
you'll see "stop cluster". Run that all services will stop cleanly. Then
check disk usage as suggested earlier.

Re: Unable to delete hosts or assign roles

gagabonar — Tue, 02 Sep 2014 12:06:06 GMT

I tried to stop the cluster as you suggested via Cloudera Manager, I got the following error:

Path: http://ipaddress:port/cmf/clusters/1/do

Version: Cloudera Enterprise Data Hub Edition Trial 5.0.2 (#297 built by jenkins on 20140606-2221 git: 80907df78ba6b50c21a598f0caff8b00685d5961)

javax.persistence.RollbackException:Error while committing the transaction
at TransactionImpl.java line 92
in org.hibernate.ejb.TransactionImpl commit()

Stack Trace:

TransactionImpl.java line 92
in org.hibernate.ejb.TransactionImpl commit()
AbstractWrappedEntityManager.java line 110
in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
CmfEntityManager.java line 368
in com.cloudera.cmf.persist.CmfEntityManager commit()
ClusterController.java line 195
in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
<generated> line -1
in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
MethodProxy.java line 191
in net.sf.cglib.proxy.MethodProxy invoke()
Cglib2AopProxy.java line 688
in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
ReflectiveMethodInvocation.java line 150
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
MethodSecurityInterceptor.java line 61
in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
ReflectiveMethodInvocation.java line 172
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
Cglib2AopProxy.java line 621
in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
<generated> line -1
in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 57
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 606
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 585
in org.springframework.web.servlet.FrameworkServlet doPost()
HttpServlet.java line 727
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 820
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 766
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 945
in org.mortbay.jetty.HttpConnection$RequestHandler content()
HttpParser.java line 756
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 218
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: javax.persistence.PersistenceException:org.hibernate.exception.GenericJDBCException: could not perform addBatch
at AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()

Stack Trace:

AbstractEntityManagerImpl.java line 1387
in org.hibernate.ejb.AbstractEntityManagerImpl convert()
AbstractEntityManagerImpl.java line 1310
in org.hibernate.ejb.AbstractEntityManagerImpl convert()
TransactionImpl.java line 80
in org.hibernate.ejb.TransactionImpl commit()
AbstractWrappedEntityManager.java line 110
in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
CmfEntityManager.java line 368
in com.cloudera.cmf.persist.CmfEntityManager commit()
ClusterController.java line 195
in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
<generated> line -1
in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
MethodProxy.java line 191
in net.sf.cglib.proxy.MethodProxy invoke()
Cglib2AopProxy.java line 688
in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
ReflectiveMethodInvocation.java line 150
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
MethodSecurityInterceptor.java line 61
in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
ReflectiveMethodInvocation.java line 172
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
Cglib2AopProxy.java line 621
in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
<generated> line -1
in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 57
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 606
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 585
in org.springframework.web.servlet.FrameworkServlet doPost()
HttpServlet.java line 727
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 820
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 766
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 945
in org.mortbay.jetty.HttpConnection$RequestHandler content()
HttpParser.java line 756
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 218
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Stack Trace:

StandardSQLExceptionConverter.java line 54
in org.hibernate.exception.internal.StandardSQLExceptionConverter convert()
SqlExceptionHelper.java line 125
in org.hibernate.engine.jdbc.spi.SqlExceptionHelper convert()
BatchingBatch.java line 114
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
BatchingBatch.java line 101
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
AbstractBatchImpl.java line 149
in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
JdbcCoordinatorImpl.java line 187
in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl getBatch()
AbstractEntityPersister.java line 3045
in org.hibernate.persister.entity.AbstractEntityPersister insert()
AbstractEntityPersister.java line 3509
in org.hibernate.persister.entity.AbstractEntityPersister insert()
EntityInsertAction.java line 88
in org.hibernate.action.internal.EntityInsertAction execute()
ActionQueue.java line 377
in org.hibernate.engine.spi.ActionQueue execute()
ActionQueue.java line 369
in org.hibernate.engine.spi.ActionQueue executeActions()
ActionQueue.java line 286
in org.hibernate.engine.spi.ActionQueue executeActions()
AbstractFlushingEventListener.java line 339
in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
DefaultFlushEventListener.java line 52
in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
SessionImpl.java line 1234
in org.hibernate.internal.SessionImpl flush()
SessionImpl.java line 404
in org.hibernate.internal.SessionImpl managedFlush()
JdbcTransaction.java line 101
in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction beforeTransactionCommit()
AbstractTransactionImpl.java line 175
in org.hibernate.engine.transaction.spi.AbstractTransactionImpl commit()
TransactionImpl.java line 75
in org.hibernate.ejb.TransactionImpl commit()
AbstractWrappedEntityManager.java line 110
in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
CmfEntityManager.java line 368
in com.cloudera.cmf.persist.CmfEntityManager commit()
ClusterController.java line 195
in com.cloudera.server.web.cmf.cluster.ClusterController dispatchClusterCommand()
<generated> line -1
in com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd invoke()
MethodProxy.java line 191
in net.sf.cglib.proxy.MethodProxy invoke()
Cglib2AopProxy.java line 688
in org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation invokeJoinpoint()
ReflectiveMethodInvocation.java line 150
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
MethodSecurityInterceptor.java line 61
in org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor invoke()
ReflectiveMethodInvocation.java line 172
in org.springframework.aop.framework.ReflectiveMethodInvocation proceed()
Cglib2AopProxy.java line 621
in org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor intercept()
<generated> line -1
in com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e65e0fcf dispatchClusterCommand()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 57
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 606
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 585
in org.springframework.web.servlet.FrameworkServlet doPost()
HttpServlet.java line 727
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 820
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 766
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 945
in org.mortbay.jetty.HttpConnection$RequestHandler content()
HttpParser.java line 756
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 218
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()

Caused by: java.sql.BatchUpdateException:Batch entry 0 insert into COMMANDS (OPTIMISTIC_LOCK_VERSION, NAME, STATE, START_INSTANT, END_INSTANT, ACTIVE, RESULT_MESSAGE, RESULT_DATA, RESULT_DATA_MIME_TYPE, RESULT_DATA_FILENAME, RESULT_DATA_PATH, RESULT_DATA_REAPED, SUCCESS, INTERNAL_STATE, ARGUMENTS, AUDITED, CLUSTER_ID, SERVICE_ID, ROLE_ID, HOST_ID, SCHEDULE_ID, PARENT_ID, COMMAND_ID) values ('0', 'Stop', 'STARTED', '1409659192831', NULL, '1', NULL, NULL, NULL, NULL, NULL, '0', '0', NULL, NULL, '0', '1', NULL, NULL, NULL, NULL, NULL, '256') was aborted. Call getNextException to see the cause.
at AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()

Stack Trace:

AbstractJdbc2Statement.java line 2598
in org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler handleError()
QueryExecutorImpl.java line 1836
in org.postgresql.core.v3.QueryExecutorImpl processResults()
QueryExecutorImpl.java line 407
in org.postgresql.core.v3.QueryExecutorImpl execute()
AbstractJdbc2Statement.java line 2737
in org.postgresql.jdbc2.AbstractJdbc2Statement executeBatch()
NewProxyPreparedStatement.java line 1723
in com.mchange.v2.c3p0.impl.NewProxyPreparedStatement executeBatch()
BatchingBatch.java line 110
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch performExecution()
BatchingBatch.java line 101
in org.hibernate.engine.jdbc.batch.internal.BatchingBatch doExecuteBatch()
AbstractBatchImpl.java line 149
in org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl execute()
JdbcCoordinatorImpl.java line 187
in org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl getBatch()
AbstractEntityPersister.java line 3045
in org.hibernate.persister.entity.AbstractEntityPersister insert()
AbstractEntityPersister.java line 3509
in org.hibernate.persister.entity.AbstractEntityPersister insert()
EntityInsertAction.java line 88
in org.hibernate.action.internal.EntityInsertAction execute()
ActionQueue.java line 377
in org.hibernate.engine.spi.ActionQueue execute()
ActionQueue.java line 369
in org.hibernate.engine.spi.ActionQueue executeActions()
ActionQueue.java line 286
in org.hibernate.engine.spi.ActionQueue executeActions()
AbstractFlushingEventListener.java line 339
in org.hibernate.event.internal.AbstractFlushingEventListener performExecutions()
DefaultFlushEventListener.java line 52
in org.hibernate.event.internal.DefaultFlushEventListener onFlush()
SessionImpl.java line 1234
in org.hibernate.internal.SessionImpl flush()
SessionImpl.java line 404
in org.hibernate.internal.SessionImpl managedFlush()
JdbcTransaction.java line 101
in org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction beforeTransactionCommit()
AbstractTransactionImpl.java line 175
in org.hibernate.engine.transaction.spi.AbstractTransactionImpl commit()
TransactionImpl.java line 75
in org.hibernate.ejb.TransactionImpl commit()
AbstractWrappedEntityManager.java line 110
in com.cloudera.enterprise.AbstractWrappedEntityManager commit()
CmfEntityManager.java line 368
in com.cloudera.cmf.persist.CmfEntityManager

Re: Unable to delete hosts or assign roles

GautamG — Tue, 02 Sep 2014 12:12:35 GMT

The database is unable to write so shutdown is failing. Please run the du
command from earlier and see which directory is using the most space

Re: Unable to delete hosts or assign roles

gagabonar — Tue, 02 Sep 2014 12:18:06 GMT

Here's what I got:

0 /var/mail
4 /var/account
4 /var/crash
4 /var/cvs
4 /var/games
4 /var/local
4 /var/nis
4 /var/opt
4 /var/preserve
4 /var/tmp
4 /var/yp
8 /var/empty
16 /var/lock
16 /var/lost+found
28 /var/db
140 /var/spool
125084 /var/cache
136760 /var/run
347180 /var/log
7221464 /var/lib

Thank you for the reply.

Re: Unable to delete hosts or assign roles

GautamG — Tue, 02 Sep 2014 13:31:35 GMT

Repeat the exercise within /var/lib. Most likely it is the event server.

Re: Unable to delete hosts or assign roles

gagabonar — Wed, 03 Sep 2014 02:45:57 GMT

Here's the result largest to smallest:

5074352 /var/lib/cloudera-service-monitor
1888312 /var/lib/cloudera-host-monitor
97444 /var/lib/cloudera-scm-server-db
47992 /var/lib/rpm
33988 /var/lib/solr
25312 /var/lib/cloudera-scm-eventserver
17720 /var/lib/zookeeper
11608 /var/lib/sqoop2
9880 /var/lib/yum
4556 /var/lib/cloudera-scm-server
4276 /var/lib/oozie
3928 /var/lib/mlocate
804 /var/lib/cloudera-scm-headlamp
240 /var/lib/hue
132 /var/lib/cloudera-scm-agent
132 /var/lib/alternatives
36 /var/lib/readahead
32 /var/lib/cas
28 /var/lib/polkit-1
24 /var/lib/nfs
20 /var/lib/rhsm
16 /var/lib/samba
16 /var/lib/pgsql
12 /var/lib/stateless
12 /var/lib/prelink
8 /var/lib/udev
8 /var/lib/postfix
8 /var/lib/plymouth
8 /var/lib/hive
8 /var/lib/hadoop-yarn
8 /var/lib/hadoop-mapreduce
8 /var/lib/hadoop-hdfs
8 /var/lib/dbus
8 /var/lib/authconfig
4 /var/lib/up2date
4 /var/lib/sqoop
4 /var/lib/spark
4 /var/lib/rsyslog
4 /var/lib/random-seed
4 /var/lib/ntp
4 /var/lib/misc
4 /var/lib/logrotate.status
4 /var/lib/impala
4 /var/lib/hive-hcatalog
4 /var/lib/hadoop-httpfs
4 /var/lib/games
4 /var/lib/fprint
4 /var/lib/flume-ng
4 /var/lib/dhclient
4 /var/lib/bigtop

I forgot to mention that before I ran into thi error, symlink in /home/var has been created to extend /var using extended disk.

Thanks for the reply.

Re: Unable to delete hosts or assign roles

gagabonar — Wed, 03 Sep 2014 03:23:17 GMT

Referring to the list, cloudera-service-monitor and cloudera-host-monitor are the two largest. Could this be related to the Time Series Storage set for cloudera-service-monitor at 10 Gb?

I cannot commit any update via cloudera manager at the moment, i tried to lower the retention then I get this error:

java.lang.IllegalStateException:Cannot commit in rollback only mode.
at Preconditions.java line 149
in com.google.common.base.Preconditions checkState()

Re: Unable to delete hosts or assign roles

gagabonar — Fri, 05 Sep 2014 10:50:50 GMT

I've solved this issue.

Cloudera Embedded DB cannot write because it is located in my /var directory (under /var/lib/cloudera-scm-server-db) which is full. I use this step-by-step solution (credits to puneethabm from google group):

1) Stop the scm service:

# service cloudera-scm-server stop

# service cloudera-scm-server-db stop

2) Copy the cloudera-scm-server-db, retaining the permissions as below:
#cd /var/lib

#cp -rp cloudera-scm-server-db /dir1/lib/

#cd /var/lib/cloudera-scm-server-db

#rm -rf data/

3) Create symlink
#cd /var/lib/cloudera-scm-server-db

#ln -s /dir1/lib/cloudera-scm-server-db/data data

4) Start services:

# service cloudera-scm-server-db start

DB initialization done.

waiting for server to start.... done

server started

# service cloudera-scm-server start

By creating symlink to disk with sufficient free space, it allows Cloudera Embedded DB to perform write transaction again.

Re: Unable to delete hosts or assign roles

Darren — Fri, 05 Sep 2014 14:47:26 GMT

Glad you solved this!

Keep in mind that when using the symlink, you may need to re-create it whenever you upgrade your cloudera-scm-server-db package in the future, since the symlink confuses the packaging code.

Thanks,
Darren