question Re: missing folders under /var/run/ambari-server in Support Questions

missing folders under /var/run/ambari-server

mike_bronson7 — Sun, 25 Feb 2018 19:00:31 GMT

we have very strange problem

we installed new ambari cluster

but we notice that the folders - stack-recommendations and bootstrap on ambari server machine , are missing

stack-recommendations and bootstrap folders must be under /var/run/ambari-server

what could be the problem ?

cd /var/run/ambari-server/
[root@master02 ambari-server]# \ls -ltr
total 4
-rw-r--r-- 1 root root  6 Feb 25 10:32 ambari-server.pid

from ambari-server log


WARN [ambari-client-thread-217] StackAdvisorCommand:304 - Error occured during stack advisor command invocation:
java.io.IOException: Cannot create /var/run/ambari-server/stack-recommendations
        at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.createRequestDirectory(StackAdvisorCommand.java:322)
        at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:289)
        at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.recommend(StackAdvisorHelper.java:113)
        at org.apache.ambari.server.controller.internal.RecommendationResourceProvider.createResources(RecommendationResourceProvider.java:92)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:298)
        at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
        at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:37)
        at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73)
        at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90)
        at org.apache.ambari.server.api.services.RecommendationService.getRecommendation(RecommendationService.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)

Re: missing folders under /var/run/ambari-server

thomas_chaniac — Mon, 26 Feb 2018 17:09:31 GMT

Hello,

With which user did you install your Ambari-Server during the ambari-server setup ? Default is root.

It looks like permission issue to me.

If you don't remember the user, check Ambari-Server logs on /var/log/ambari-server/ambari-server.log.

When you found it, run chown -R <AMBARI-USER> /var/run/ambari-server

Tell us if it works better after.

Thomas

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Mon, 26 Feb 2018 17:45:34 GMT

yes , but only change the owner back to ambari is a partial solution , because we have the missing folder - stack-recommendations and bootstrap , so need to find the complete solution how to recreate this folders

Re: missing folders under /var/run/ambari-server

jsensharma — Mon, 26 Feb 2018 17:58:33 GMT

@Michael Bronson

When we try to run ambari-server as a Non Root user then it is required that this user should have Read & Write access to the following directories.

pid.dir=/var/run/ambari-server
bootstrap.dir=/var/run/ambari-server/bootstrap
recommendations.dir=/var/run/ambari-server/stack-recommendations

.
Above 3 directories default values can be found inside the "/etc/ambari-server/conf/ambari.properties"

All the permission changes are done when we run the "ambari-server setup" command which runs as root.
When we run "ambari-server setup" then ambari will create the directories with the correct permissions

Re: missing folders under /var/run/ambari-server

jsensharma — Mon, 26 Feb 2018 18:22:43 GMT

@Michael Bronson

In some cases, we might see that once we reboot the ambari server host then this directory is getting created again with "root:root" permissions.

This can happen, if by any chance if the "pid.dir" or "recommendations.dir" is mounted on "tmpfs" then rebooting the host might alter the permissions back to 'root:root' and hence the non root ambari user will again not be able to write to these directories.

Please NOTE that if you face such issue of getting the permission changed for these directories then please change the values "pid.dir", "bootstrap.dir" and "recommendations.dir" insife ambari.properties file to point to a directory which is not on tmpfs. Like "/var/ambari-server" shouldn't be deleted by the operating system after host reboot. So in such scenarios moving the directories under non tmpfs and re-running ambari-server setup is the best option.

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Mon, 26 Feb 2018 18:53:41 GMT

<br>

@Jay thank you so much for the solution

just one question - can we be sure that set the folders to be under /var and not under /var/run , not

Damage to ambari cluster functionality ?

first I will change the code in the ambari Kit in order to update ambari.properties file per your great recommendation

so bootstrap.dir , pid.dir , recommendations.dir will be under /var and not under /var/run (this is for scratch installation)

# grep run /etc/ambari-server/conf/ambari.properties

bootstrap.dir=/var/ambari-server/bootstrap
pid.dir=/var/ambari-server
recommendations.dir=/var/ambari-server/stack-recommendations<br>

in order to fix the current ambari cluster we will change ambari.properties the same as below ( scratch installation ) , and we will run ambari setup to recreate the missing folders

please just approve my steps

Re: missing folders under /var/run/ambari-server

jsensharma — Mon, 26 Feb 2018 19:00:55 GMT

@Michael Bronson

Yes, your steps looks correct:

0. Stop ambari-server

# ambari-server stop

1. Change the PATH for those 3 properties mentioned in the ambari.properties file.

pid.dir=/var/ambari-server
bootstrap.dir=/var/ambari-server/bootstrap
recommendations.dir=/var/ambari-server/stack-recommendations<br>

2. Run the "ambari-server setup" and during the ambari-server setup process, when prompted to Customize user account for ambari-server daemon?, choose y
(Please do not change any other properties)

NOTE: Just to avoid any mistakes it is always better to take ambari DB dump (just to make sure that by mistake we do not make any other db changes)

3. Then restart ambari-server.

# ambari-server start

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Mon, 12 Mar 2018 21:21:31 GMT

@Dear Jay

We performed the following "procedure" per you recommendation

Seems that procedure completed successfully but still stack-recommendations folder not created under /var/ambari-server

where we are wrong?

we change the following in ambari.properties before the procedure :

pid.dir=/var/run/ambari-server
bootstrap.dir=/var/run/ambari-server/bootstrap
recommendations.dir=/var/run/ambari-server/stack-recommendations

The procedure:

NOTE - you mentioned that - "when prompted to Customize user account for ambari-server daemon?, choose y"

but this question not appears during the process

<br>[root@master02 ~]# ambari-server stop
Using python  /usr/bin/python
Stopping ambari-server
Waiting for server stop...
Ambari Server stopped


[root@master02 ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Ambari-server daemon is configured to run under user 'ambari'. Change this setting [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)?
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)?
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.


[root@master02 ~]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................................
Server started listening on 8080
DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
[root@master02 ~]# ls -ltr /var/ambari-server/bootstrap/
total 0


[root@master02 ~]# ls -ltr /var/ambari-server/
total 4

drwxr-xr-x 2 ambari root 6 Mar 12 14:10 bootstrap
-rw-r--r-- 1 root   root 6 Mar 12 14:10 ambari-server.pid


[root@master02 ~]# find /var -name  stack-recommendations
[root@master02 ~]#

Re: missing folders under /var/run/ambari-server

jsensharma — Tue, 13 Mar 2018 05:52:48 GMT

@Michael Bronson

Please try making any small config change from ambari UI and then see that the dir "stack-recommendations" created or not? If there is nothing to recommend then you won't see the dir created.

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Tue, 13 Mar 2018 12:52:07 GMT

@Jay , ok but do you have any API suggestion to perform some refresh or reload in order to create the folder , because we are working on production clusters and we not want to change something that not necessary

Re: missing folders under /var/run/ambari-server

jsensharma — Tue, 13 Mar 2018 13:48:04 GMT

@Michael Bronson

we do not need to worry about this "stack-recommendations" directory it should be created as soon as we make any configuration changes to ambari UI and when ambari has some recommendations for us.

But if you want to test before hand then you might want to refer to the following HCC thread which suggexst how you can run some stack advisory scripts on your own. (i have not tested them manually on my own)

https://community.hortonworks.com/questions/141855/stack-advisor-how-to-use-it.html

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Tue, 13 Mar 2018 15:21:35 GMT

@Jay indeed when I change some value from ambari then I saw the folder - /var//ambari-server/stack-recommendations , but after that we need to return the value back as was , but this approach require us the following steps ( from script )


1. change some value from ambari GUI ( by API )
2 because change we need to restart relevant services ( by API )
3 change back the value as was before ( by API )
4 second restart services to complete ( by API )

as you see from steps 1-4 all this are needed only to create the stack-recommendations folder

I want to share with you the all picture

"we complete to create auto tool that perform ambari upgrade

during the ambari upgrade auto process it create ambari dump (pg_dump -U ambari -f /var/tmp/dbdumps/ambari.sql)

but this dump required stack-recommendations folder

so in case stack-recommendations isnt under /var/run then the script will update the ambari properties and re-run the ambari setup in order to prepare the folder under /var/ambari-server , and then script need to run API to change value in order to create the stack-recommendations folder and do restart ......."

so this approach required too many steps in order to bring the folder live

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Tue, 13 Mar 2018 17:34:06 GMT

@Jay regarding to my previeos comment , do you have other simple solution ( instead steps 1-4) ?

Re: missing folders under /var/run/ambari-server

jsensharma — Wed, 14 Mar 2018 04:58:02 GMT

@Michael Bronson

What i wanted to say is that if we have setup the "pid.dir" , "bootstrap.dir" , "recommendations.dir" outside the "tmpfs" mount and the non root ambari user has proper read/write permission to these directories then we do not need to worry about these directoies anymore. Whenever we make any changes to these directories only once we will need to run the "ambari-server setup" command just to ensure that ambari creates some of these directories (not necesarily all immediately) like "recommendations.dir" ambari might create later as well.

Ambari will create those directories on it's own. We do not need to run "ambari-serevr setup" command many times.

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Wed, 14 Mar 2018 13:36:39 GMT

Dear Jay thank you so much for all explains ,

just one last idea that I was thinking about this, in order to enable the ambari-server backup

please tell me please you opinion?

as you see here when we backup the ambari we get error because the missing folders

/usr/sbin/ambari-server backup
Using python  /usr/bin/python
Backing up Ambari File System state... *this will not backup the server database*
Backup requested.
No path specified. Will use /var/lib/ambari-server/Ambari_State_Backup.zip
Backup process initiated.
Error while validating folders. Folder /var/run/ambari-server/bootstrap/ does not exist.

so as workaround I create empty folders as:

mkdir /var/run/ambari-server/stack-recommendations
mkdir /var/run/ambari-server/bootstrap/

now we run again the ambari-server backup , and backup succeeded

 /usr/sbin/ambari-server backup
Using python  /usr/bin/python
Backing up Ambari File System state... *this will not backup the server database*
Backup requested.
No path specified. Will use /var/lib/ambari-server/Ambari_State_Backup.zip
Backup process initiated.
Creating zip file...
Zip file created at /var/lib/ambari-server/Ambari_State_Backup.zip
Backup complete.
Ambari Server 'backup' completed successfully.

Jay what you think about this approach ?

Re: missing folders under /var/run/ambari-server

jsensharma — Wed, 14 Mar 2018 14:03:17 GMT

@Michael Bronson

Yes, that is correct that the "ambar-server backup" utility will check if the "stack-recommendation" directory exists or not?

If this directory does not exist then we will need to manually create that directory so that it passes the check. That is a workaround.

https://github.com/apache/ambari/blob/release-2.6.1/ambari-server/src/main/python/ambari_server/BackupRestore.py#L131-L139

the following 4 directories it checks:

# The list of files where the ambari server state is kept on the filesystem
AMBARI_FILESYSTEM_STATE = [AmbariPath.get("/etc/ambari-server/conf"),
                           AmbariPath.get("/var/lib/ambari-server/resources"),
                           AmbariPath.get("/var/run/ambari-server/bootstrap/"),
                           AmbariPath.get("/var/run/ambari-server/stack-recommendations")]

In normal cases when the cluster is running for quite some time and some configuration changes have been made to the cluster then definitely we will see some contents inside this directory. even when the ambari is running as non root user.

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Wed, 14 Mar 2018 14:17:35 GMT

ok Jay , so I will do the workaround in order to enable the ambari-server backup ( create empty folders - stack-recommendations , bootstrap)

Re: missing folders under /var/run/ambari-server

jsensharma — Wed, 14 Mar 2018 14:23:24 GMT

@Michael Bronson

Yes, if ambari does not create them and we want to run the backup command then we can create the directories manually. It should be Ok.

Re: missing folders under /var/run/ambari-server

mike_bronson7 — Wed, 14 Mar 2018 14:35:15 GMT

@Jay again thasnk you so much yours remarks are very useful to us , and your knowledge is empresive

Re: missing folders under /var/run/ambari-server

jsensharma — Wed, 14 Mar 2018 14:37:45 GMT

@Michael Bronson

Thank you for appreciation and i apologies for making this thread a bit confusing in few of my previous update 😉