Can't start yarn after cleaning /tmp

Cico — Wed, 16 Oct 2019 18:12:11 GMT

Searching on internet I found that I can clean the tmp directory without problem on a cloudera cluster, so I did it.

Now, when I try to start the cluster all services goes up well, but yarn failed to start with this error:

19/10/16 15:50:17 FATAL nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start. at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:79) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:329) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:563) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609) Caused by: org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279) at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:65) ... 6 more Caused by: java.io.IOException: Unable to initialize WebAppContext at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:922) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274) ... 7 more Caused by: java.io.FileNotFoundException: /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp/webapps/node/.keep (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at org.mortbay.resource.JarResource.extract(JarResource.java:215) at org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974) at org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832) at org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:900)

What can I do?

Re: Can't start yarn after cleaning /tmp

paras — Thu, 17 Oct 2019 08:24:01 GMT

@Cico

Error indicates that the yarn user is unable to access the file /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp/webapps/node/.keep

It seems there is an issue with /tmp directory permissions on local.

Login to the particular Nodemanager node which reports the error and check if /tmp directory permissions are set to 777. This should allow yarn user to access the file

# chmod 777 /tmp

Re: Can't start yarn after cleaning /tmp

Cico — Thu, 17 Oct 2019 08:27:38 GMT

There is no file Jetty_0_0_0_0_8042_node in /tmp, the partition is been wiped out.

Is not a problem of permission, but of missing file.

How can I let yarn to remake those files?

Re: Can't start yarn after cleaning /tmp

paras — Thu, 17 Oct 2019 08:40:01 GMT

@Cico

If the permissions are already set to 777 for /tmp , restart the particular node manager from Cloudera Manager. This should allow the creation of new file in /tmp.

Re: Can't start yarn after cleaning /tmp

Cico — Thu, 17 Oct 2019 08:45:57 GMT

Thank you very much!

You saved the day.

question Re: Can't start yarn after cleaning /tmp in Support Questions

Can't start yarn after cleaning /tmp

Re: Can't start yarn after cleaning /tmp

Re: Can't start yarn after cleaning /tmp

Re: Can't start yarn after cleaning /tmp

Re: Can't start yarn after cleaning /tmp