Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problem starting nodemanager

Problem starting nodemanager

Contributor

Hello all

 

I could not start YARN service on my fresh installation.

 

I followed following URL for the installation and I am using cm-5.4.3

 

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_ig_install_path_c.ht...

Installation Path C - Manual Installation Using Cloudera Manager Tarballs

 

I am using one node for startup.

 

I got following errors while startup YARN first time.

 

 

1.jpg

 

 

 

Details screen

 

2.jpg

more details

 

3.jpg

 

stderr output

 

 

4.jpg

 

 

stdout output

 

5.jpg

 

 

Role log details

 

6.jpg

 

 

 

my /var/log/directory has link to /opt/cloudera/log/cloudera-scm-server directory as follows

 

7.jpg

 

I could create a file in cloudera-scm-server's log directory as cloudera-scm user (I changed the user's login capabilities for testing)

 

8.jpg

 

A python process listening 9000/TCP

 

9.jpg

 

/opt/cloudera/lib and /opt/cloudera/log directories have following directories 

 

 

10.jpg

 

I could also have read/write option on hadoop-yarn directory as yarn user, I tested it

 

I changed /tmp with 

mount -o remount exec /tmp 

No progress, then I revert it to original state

 

So how I install and start YARN and Nodemanager

 

PS:Our Unix team has some standarts about FS sizing, therefore I create my own log and lib directories and I linked into /var/lib and /var/log as subdir.

 

Best Regards

Murat

 

 

 

13 REPLIES 13

Re: Problem starting nodemanager

Explorer

Are you creating a local cluster? or using AWS/GCE?

Re: Problem starting nodemanager

Contributor
Hello rajnandy

Thanks for your input

I am installing into local cluster

Best Regards
Murat

Re: Problem starting nodemanager

Explorer

Can you please attach the latest log in the cloudera-scm-manager dir? atleast last 200 lines with the error.

Re: Problem starting nodemanager

Contributor

Hello 

 

I tried to add last 250 line from /opt/cloudera/cloudera-manager/cm-5.4.3/log/cloudera-scm-server/cloudera-scm-server.out but it exceeds 50.000 chars of forum limit. 

So I tried to upload the file into dropbox. You could get it from following link

 

Dropbox Document Address

 

If you could not get from dropbox, please let me know

 

Thanks for your all responses 

Murat

Re: Problem starting nodemanager

Explorer

Thanks for the logs. I can see the following error in the log:

ConnectException: Connection refused to http://localhost:8080/manifest.json

 

Can you please make sure

1. selinux is disabled 

2. iptables is turned off

in the cluster nodes.

You can simply google for these two.

 

 

Also please send the content of the /etc/hosts

Re: Problem starting nodemanager

Contributor

 Hello 

 

Thanks for your quick response

 

1--> selinux disabled

2--> all of the 3 iptables policy is accept, there is no deny rule in iptables  anyway I shutdown the iptables  and then tried to start again. Again no progress.

I added another dropbox link for log file update

 

Dropbox link

 

I am so sorry but could not share hosts file.

There is only one node in cluster for initial phase, I will add other nodes later.

But I could say that the node ping itself with ip addresses, hostname and also Fully Qualified Domain Name. So there is no problem in ip addresses<-> hostname/FQDN resolution

What kind of issue in hosts file we are looking for?

 

Re: Problem starting nodemanager

Explorer

Yes, I was looking for FQDN issues only. But you said those are working fine. There must be some firewall blocking 8080 port.

Are you abke to telnet to the following port with localhost and hostname?

Re: Problem starting nodemanager

Contributor
Hello again

I got "connection refused" from my telnet <localhost|hostname|FQDN|ip addresses 8080> requests even iptables down

Thanks

Re: Problem starting nodemanager

Contributor

Hello again

Regarding to "http://localhost:8080/manifest.json" error, when I was installing cloudera from parcels, installation application cloud not find parcels in /opt/cloudera/parcels directory. When I was searching the issue in cloudera.com, I found a solution about it.

 

Solution says that 

  • copy all the parcels into another directory,
  • change directory to this directory
  • enter following command to access parcels from also this directory  " python -m SimpleHTTPServer 8080"
  • reconfigure installation gui to search parcels in localhost:8080

so I restarted that simple HTTP server and then tried to start YARN Nodemanager, then I got following logs in  cloudera-scm-server.log

 

I forgot to mentioned this HTTP serving information in  previous posts, I am so sorry

 

Thanks

Murat

 

2015-07-21 11:45:56,415 INFO 1074097702@scm-web-1502:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command YarnOrderedStart SvcCmdArgs{targetRoles=[DbRole{id=38, name=yarn-RESOURCEMANAGER-414d2e126480919999b94a89074ed3f4, hostName=nahdp1.vodafone.local}, DbRole{id=37, name=yarn-JOBHISTORY-414d2e126480919999b94a89074ed3f4, hostName=nahdp1.vodafone.local}, DbRole{id=36, name=yarn-NODEMANAGER-414d2e126480919999b94a89074ed3f4, hostName=nahdp1.vodafone.local}], args=[]}. Service: DbService{id=17, name=yarn}
2015-07-21 11:45:56,425 INFO 1074097702@scm-web-1502:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command Start SvcCmdArgs{targetRoles=[DbRole{id=37, name=yarn-JOBHISTORY-414d2e126480919999b94a89074ed3f4, hostName=nahdp1.vodafone.local}, DbRole{id=36, name=yarn-NODEMANAGER-414d2e126480919999b94a89074ed3f4, hostName=nahdp1.vodafone.local}], args=[]}. Service: DbService{id=17, name=yarn}
2015-07-21 11:45:56,443 INFO 1074097702@scm-web-1502:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing role command Start BasicCmdArgs{args=[]}. Service: DbService{id=17, name=yarn} Role: DbRole{id=36, name=yarn-NODEMANAGER-414d2e126480919999b94a89074ed3f4, hostName=nahdp1.vodafone.local}
2015-07-21 11:45:56,482 INFO 1074097702@scm-web-1502:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: Added BringUp command to service DbService{id=17, name=yarn}.
2015-07-21 11:45:56,538 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (148) is still active on service DbService{id=17, name=yarn}.
2015-07-21 11:46:01,550 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (148) is still active on service DbService{id=17, name=yarn}.
2015-07-21 11:46:06,563 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (148) is still active on service DbService{id=17, name=yarn}.
2015-07-21 11:46:11,577 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (148) is still active on service DbService{id=17, name=yarn}.
2015-07-21 11:46:11,673 INFO 51492618@agentServer-0:com.cloudera.cmf.command.components.StalenessChecker: No staleness check scheduled, scheduling one in 30 seconds
2015-07-21 11:46:11,704 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (149) has finished unsuccessfully on service DbService{id=17, name=yarn} role DbRole{id=36, name=yarn-NODEMANAGER-414d2e126480919999b94a89074ed3f4, hostName=nahdp1.vodafone.local}.
2015-07-21 11:46:11,707 INFO CommandPusher:com.cloudera.cmf.service.AbstractBringUpBringDownCommands: BringUp command (148) has finished on service DbService{id=17, name=yarn}.
2015-07-21 11:46:41,674 INFO ScheduledStalenessChecker:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[First reason why: Process (id=60) has a brand new heartbeat]}.
2015-07-21 11:46:41,888 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT0.204S
2015-07-21 11:52:34,401 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2015-07-21 11:52:34,402 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2013-07-21T08:52:34.401Z to reap.
2015-07-21 11:52:34,403 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeScannerService: Reaped 0 requests.
2015-07-21 11:52:34,403 INFO StaleEntityEviction:com.cloudera.server.cmf.node.NodeConfiguratorService: Reaped 0 requests.
2015-07-21 11:54:19,748 WARN 2040558817@scm-web-1511:com.cloudera.server.web.cmf.StatusProvider: (123 skipped) Failed to get status.
com.cloudera.cmon.MgmtServiceNotRunningException: HOST_MONITORING is not running
at com.cloudera.cmon.MgmtServiceLocator.getNozzleIPC(MgmtServiceLocator.java:145)
at com.cloudera.server.web.cmf.StatusProvider.getResponseCallable(StatusProvider.java:500)
at com.cloudera.server.web.cmf.StatusProvider.getStatus(StatusProvider.java:448)
at com.cloudera.server.web.cmf.AggregateStatusController$HealthInfo.<init>(AggregateStatusController.java:743)
at com.cloudera.server.web.cmf.AggregateStatusController$HealthInfo.<init>(AggregateStatusController.java:711)
at com.cloudera.server.web.cmf.AggregateStatusController$2.call(AggregateStatusController.java:1263)
at com.cloudera.server.web.cmf.AggregateStatusController$2.call(AggregateStatusController.java:1260)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4792)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4789)
at com.cloudera.server.web.cmf.AggregateStatusController.getHealthInfo(AggregateStatusController.java:1258)
at com.cloudera.server.web.cmf.AggregateStatusController.servicesTable(AggregateStatusController.java:413)
at sun.reflect.GeneratedMethodAccessor1221.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:585)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2015-07-21 11:57:07,987 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (29 skipped) Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy120.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:312)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:299)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:135)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.

Don't have an account?
Coming from Hortonworks? Activate your account here