missing folders under /var/run/ambari-server

[#1]

we have very strange problem

we installed new ambari cluster

but we notice that the folders - stack-recommendations and bootstrap on ambari server machine , are missing

stack-recommendations and bootstrap folders must be under /var/run/ambari-server

what could be the problem ?

cd /var/run/ambari-server/
[root@master02 ambari-server]# \ls -ltr
total 4
-rw-r--r-- 1 root root  6 Feb 25 10:32 ambari-server.pid

from ambari-server log


WARN [ambari-client-thread-217] StackAdvisorCommand:304 - Error occured during stack advisor command invocation:
java.io.IOException: Cannot create /var/run/ambari-server/stack-recommendations
        at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.createRequestDirectory(StackAdvisorCommand.java:322)
        at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:289)
        at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.recommend(StackAdvisorHelper.java:113)
        at org.apache.ambari.server.controller.internal.RecommendationResourceProvider.createResources(RecommendationResourceProvider.java:92)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:298)
        at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
        at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:37)
        at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73)
        at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90)
        at org.apache.ambari.server.api.services.RecommendationService.getRecommendation(RecommendationService.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)

[#2]

@Michael Bronson

we do not need to worry about this "stack-recommendations" directory it should be created as soon as we make any configuration changes to ambari UI and when ambari has some recommendations for us.

But if you want to test before hand then you might want to refer to the following HCC thread which suggexst how you can run some stack advisory scripts on your own. (i have not tested them manually on my own)

https://community.hortonworks.com/questions/141855/stack-advisor-how-to-use-it.html

[#3]

@Jay indeed when I change some value from ambari then I saw the folder - /var//ambari-server/stack-recommendations , but after that we need to return the value back as was , but this approach require us the following steps ( from script )

1. change some value from ambari GUI ( by API )
2 because change we need to restart relevant services ( by API )
3 change back the value as was before ( by API )
4 second restart services to complete ( by API )

as you see from steps 1-4 all this are needed only to create the stack-recommendations folder

I want to share with you the all picture

"we complete to create auto tool that perform ambari upgrade

during the ambari upgrade auto process it create ambari dump (pg_dump -U ambari -f /var/tmp/dbdumps/ambari.sql)

but this dump required stack-recommendations folder

so in case stack-recommendations isnt under /var/run then the script will update the ambari properties and re-run the ambari setup in order to prepare the folder under /var/ambari-server , and then script need to run API to change value in order to create the stack-recommendations folder and do restart ......."

so this approach required too many steps in order to bring the folder live

[#4]

@Michael Bronson

What i wanted to say is that if we have setup the "pid.dir" , "bootstrap.dir" , "recommendations.dir" outside the "tmpfs" mount and the non root ambari user has proper read/write permission to these directories then we do not need to worry about these directoies anymore. Whenever we make any changes to these directories only once we will need to run the "ambari-server setup" command just to ensure that ambari creates some of these directories (not necesarily all immediately) like "recommendations.dir" ambari might create later as well.

Ambari will create those directories on it's own. We do not need to run "ambari-serevr setup" command many times.

.

[#5]

@Jay regarding to my previeos comment , do you have other simple solution ( instead steps 1-4) ?

[#6]

Dear Jay thank you so much for all explains ,

just one last idea that I was thinking about this, in order to enable the ambari-server backup

please tell me please you opinion?

as you see here when we backup the ambari we get error because the missing folders

/usr/sbin/ambari-server backup
Using python  /usr/bin/python
Backing up Ambari File System state... *this will not backup the server database*
Backup requested.
No path specified. Will use /var/lib/ambari-server/Ambari_State_Backup.zip
Backup process initiated.
Error while validating folders. Folder /var/run/ambari-server/bootstrap/ does not exist.

so as workaround I create empty folders as:

mkdir /var/run/ambari-server/stack-recommendations
mkdir /var/run/ambari-server/bootstrap/

now we run again the ambari-server backup , and backup succeeded

 /usr/sbin/ambari-server backup
Using python  /usr/bin/python
Backing up Ambari File System state... *this will not backup the server database*
Backup requested.
No path specified. Will use /var/lib/ambari-server/Ambari_State_Backup.zip
Backup process initiated.
Creating zip file...
Zip file created at /var/lib/ambari-server/Ambari_State_Backup.zip
Backup complete.
Ambari Server 'backup' completed successfully.

Jay what you think about this approach ?

[#7]

@Michael Bronson

Yes, that is correct that the "ambar-server backup" utility will check if the "stack-recommendation" directory exists or not?

If this directory does not exist then we will need to manually create that directory so that it passes the check. That is a workaround.

https://github.com/apache/ambari/blob/release-2.6.1/ambari-server/src/main/python/ambari_server/Back...

the following 4 directories it checks:

# The list of files where the ambari server state is kept on the filesystem
AMBARI_FILESYSTEM_STATE = [AmbariPath.get("/etc/ambari-server/conf"),
                           AmbariPath.get("/var/lib/ambari-server/resources"),
                           AmbariPath.get("/var/run/ambari-server/bootstrap/"),
                           AmbariPath.get("/var/run/ambari-server/stack-recommendations")]

.

In normal cases when the cluster is running for quite some time and some configuration changes have been made to the cluster then definitely we will see some contents inside this directory. even when the ambari is running as non root user.

[#8]

ok Jay , so I will do the workaround in order to enable the ambari-server backup ( create empty folders - stack-recommendations , bootstrap)

[#9]

@Michael Bronson

Yes, if ambari does not create them and we want to run the backup command then we can create the directories manually. It should be Ok.

[#10]

@Jay again thasnk you so much yours remarks are very useful to us , and your knowledge is empresive

[#11]

@Michael Bronson

Thank you for appreciation and i apologies for making this thread a bit confusing in few of my previous update 😉