Reply
Highlighted
Explorer
Posts: 7
Registered: ‎11-16-2016

HDFS File Browser issue

hello 

 

Today we want to change the host (OS:centos7) IP address.

 

We find we can not restart the CDH,so we recover the IP address and then  we see the error in HDFS

 

.

 

Version: Cloudera Enterprise Data Hub Edition Trial 5.4.3 (#258 built by jenkins on 20150625-2046 git: 1139ffb081360fbbb1be8a19a87ca3c52ee4b1cf)
com.cloudera.headlamp.api.HeadlampOperationException:
at HeadlampService.java line 12315
in com.cloudera.headlamp.api.HeadlampService$listWatchedDirectories_result read()
Stack Trace:

HeadlampService.java line 12315
in com.cloudera.headlamp.api.HeadlampService$listWatchedDirectories_result read()
TServiceClient.java line 78
in org.apache.thrift.TServiceClient receiveBase()
HeadlampService.java line 563
in com.cloudera.headlamp.api.HeadlampService$Client recv_listWatchedDirectories()
HeadlampService.java line 550
in com.cloudera.headlamp.api.HeadlampService$Client listWatchedDirectories()
BrowsePageModel.java line 60
in com.cloudera.server.web.headlamp.hdfs.BrowsePageModel getWatchedDirs()
BrowsePageModel.java line 48
in com.cloudera.server.web.headlamp.hdfs.BrowsePageModel getWatchedDirs()
ServiceBrowseController.java line 132
in com.cloudera.server.web.cmf.ServiceBrowseController browseHdfs()
ServiceBrowseController.java line 108
in com.cloudera.server.web.cmf.ServiceBrowseController getBrowser()
NativeMethodAccessorImpl.java line -2
in sun.reflect.NativeMethodAccessorImpl invoke0()
NativeMethodAccessorImpl.java line 57
in sun.reflect.NativeMethodAccessorImpl invoke()
DelegatingMethodAccessorImpl.java line 43
in sun.reflect.DelegatingMethodAccessorImpl invoke()
Method.java line 606
in java.lang.reflect.Method invoke()
HandlerMethodInvoker.java line 176
in org.springframework.web.bind.annotation.support.HandlerMethodInvoker invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 436
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter invokeHandlerMethod()
AnnotationMethodHandlerAdapter.java line 424
in org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter handle()
DispatcherServlet.java line 790
in org.springframework.web.servlet.DispatcherServlet doDispatch()
DispatcherServlet.java line 719
in org.springframework.web.servlet.DispatcherServlet doService()
FrameworkServlet.java line 669
in org.springframework.web.servlet.FrameworkServlet processRequest()
FrameworkServlet.java line 574
in org.springframework.web.servlet.FrameworkServlet doGet()
HttpServlet.java line 707
in javax.servlet.http.HttpServlet service()
HttpServlet.java line 820
in javax.servlet.http.HttpServlet service()
ServletHolder.java line 511
in org.mortbay.jetty.servlet.ServletHolder handle()
ServletHandler.java line 1221
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
UserAgentFilter.java line 78
in org.mortbay.servlet.UserAgentFilter doFilter()
GzipFilter.java line 131
in org.mortbay.servlet.GzipFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JAMonServletFilter.java line 48
in com.jamonapi.http.JAMonServletFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
JavaMelodyFacade.java line 109
in com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
FilterChainProxy.java line 311
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterSecurityInterceptor.java line 116
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor invoke()
FilterSecurityInterceptor.java line 83
in org.springframework.security.web.access.intercept.FilterSecurityInterceptor doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ExceptionTranslationFilter.java line 113
in org.springframework.security.web.access.ExceptionTranslationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SessionManagementFilter.java line 101
in org.springframework.security.web.session.SessionManagementFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AnonymousAuthenticationFilter.java line 113
in org.springframework.security.web.authentication.AnonymousAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RememberMeAuthenticationFilter.java line 146
in org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextHolderAwareRequestFilter.java line 54
in org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
RequestCacheAwareFilter.java line 45
in org.springframework.security.web.savedrequest.RequestCacheAwareFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
AbstractAuthenticationProcessingFilter.java line 182
in org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
LogoutFilter.java line 105
in org.springframework.security.web.authentication.logout.LogoutFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
SecurityContextPersistenceFilter.java line 87
in org.springframework.security.web.context.SecurityContextPersistenceFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
ConcurrentSessionFilter.java line 125
in org.springframework.security.web.session.ConcurrentSessionFilter doFilter()
FilterChainProxy.java line 323
in org.springframework.security.web.FilterChainProxy$VirtualFilterChain doFilter()
FilterChainProxy.java line 173
in org.springframework.security.web.FilterChainProxy doFilter()
DelegatingFilterProxy.java line 237
in org.springframework.web.filter.DelegatingFilterProxy invokeDelegate()
DelegatingFilterProxy.java line 167
in org.springframework.web.filter.DelegatingFilterProxy doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
CharacterEncodingFilter.java line 88
in org.springframework.web.filter.CharacterEncodingFilter doFilterInternal()
OncePerRequestFilter.java line 76
in org.springframework.web.filter.OncePerRequestFilter doFilter()
ServletHandler.java line 1212
in org.mortbay.jetty.servlet.ServletHandler$CachedChain doFilter()
ServletHandler.java line 399
in org.mortbay.jetty.servlet.ServletHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
SessionHandler.java line 182
in org.mortbay.jetty.servlet.SessionHandler handle()
SecurityHandler.java line 216
in org.mortbay.jetty.security.SecurityHandler handle()
ContextHandler.java line 767
in org.mortbay.jetty.handler.ContextHandler handle()
WebAppContext.java line 450
in org.mortbay.jetty.webapp.WebAppContext handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
StatisticsHandler.java line 53
in org.mortbay.jetty.handler.StatisticsHandler handle()
HandlerWrapper.java line 152
in org.mortbay.jetty.handler.HandlerWrapper handle()
Server.java line 326
in org.mortbay.jetty.Server handle()
HttpConnection.java line 542
in org.mortbay.jetty.HttpConnection handleRequest()
HttpConnection.java line 928
in org.mortbay.jetty.HttpConnection$RequestHandler headerComplete()
HttpParser.java line 549
in org.mortbay.jetty.HttpParser parseNext()
HttpParser.java line 212
in org.mortbay.jetty.HttpParser parseAvailable()
HttpConnection.java line 404
in org.mortbay.jetty.HttpConnection handle()
SelectChannelEndPoint.java line 410
in org.mortbay.io.nio.SelectChannelEndPoint run()
QueuedThreadPool.java line 582
in org.mortbay.thread.QueuedThreadPool$PoolThread run()

 

 

how can we fix it?

Posts: 642
Topics: 3
Kudos: 118
Solutions: 67
Registered: ‎08-16-2016

Re: HDFS File Browser issue

So you changed the IP address and now you cannot restart any services on the cluster.

Check the Hosts section in CM. It should have the old IP addresses and this is likely were the issue is coming from as the hostname and IP address are written to the CM database when the CM agent is registered with the cluster.

If this is the case, you can try finding all entries of the IP address in the CM DB and updating it or you can remove and re-add the CM agent. The latter would need each node to have all roles removed from it to remove it from the cluster and then delete it from CM.
Announcements