Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

missing folders under /var/run/ambari-server

avatar

we have very strange problem

we installed new ambari cluster

but we notice that the folders - stack-recommendations and bootstrap on ambari server machine , are missing

stack-recommendations and bootstrap folders must be under /var/run/ambari-server

what could be the problem ?

cd /var/run/ambari-server/
[root@master02 ambari-server]# \ls -ltr
total 4
-rw-r--r-- 1 root root  6 Feb 25 10:32 ambari-server.pid
from ambari-server log


WARN [ambari-client-thread-217] StackAdvisorCommand:304 - Error occured during stack advisor command invocation:
java.io.IOException: Cannot create /var/run/ambari-server/stack-recommendations
        at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.createRequestDirectory(StackAdvisorCommand.java:322)
        at org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:289)
        at org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.recommend(StackAdvisorHelper.java:113)
        at org.apache.ambari.server.controller.internal.RecommendationResourceProvider.createResources(RecommendationResourceProvider.java:92)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:298)
        at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
        at org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:37)
        at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73)
        at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90)
        at org.apache.ambari.server.api.services.RecommendationService.getRecommendation(RecommendationService.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
Michael-Bronson
1 ACCEPTED SOLUTION

avatar
Master Mentor

@Michael Bronson

When we try to run ambari-server as a Non Root user then it is required that this user should have Read & Write access to the following directories.

pid.dir=/var/run/ambari-server
bootstrap.dir=/var/run/ambari-server/bootstrap
recommendations.dir=/var/run/ambari-server/stack-recommendations

.
Above 3 directories default values can be found inside the "/etc/ambari-server/conf/ambari.properties"

All the permission changes are done when we run the "ambari-server setup" command which runs as root.
When we run "ambari-server setup" then ambari will create the directories with the correct permissions

.

View solution in original post

19 REPLIES 19

avatar
Contributor

Hello,

With which user did you install your Ambari-Server during the ambari-server setup ? Default is root.

It looks like permission issue to me.

If you don't remember the user, check Ambari-Server logs on /var/log/ambari-server/ambari-server.log.

When you found it, run chown -R <AMBARI-USER> /var/run/ambari-server

Tell us if it works better after.

Thomas

avatar

yes , but only change the owner back to ambari is a partial solution , because we have the missing folder - stack-recommendations and bootstrap , so need to find the complete solution how to recreate this folders

Michael-Bronson

avatar
Master Mentor

@Michael Bronson

When we try to run ambari-server as a Non Root user then it is required that this user should have Read & Write access to the following directories.

pid.dir=/var/run/ambari-server
bootstrap.dir=/var/run/ambari-server/bootstrap
recommendations.dir=/var/run/ambari-server/stack-recommendations

.
Above 3 directories default values can be found inside the "/etc/ambari-server/conf/ambari.properties"

All the permission changes are done when we run the "ambari-server setup" command which runs as root.
When we run "ambari-server setup" then ambari will create the directories with the correct permissions

.

avatar
Master Mentor

@Michael Bronson

In some cases, we might see that once we reboot the ambari server host then this directory is getting created again with "root:root" permissions.

This can happen, if by any chance if the "pid.dir" or "recommendations.dir" is mounted on "tmpfs" then rebooting the host might alter the permissions back to 'root:root' and hence the non root ambari user will again not be able to write to these directories.

Please NOTE that if you face such issue of getting the permission changed for these directories then please change the values "pid.dir", "bootstrap.dir" and "recommendations.dir" insife ambari.properties file to point to a directory which is not on tmpfs. Like "/var/ambari-server" shouldn't be deleted by the operating system after host reboot. So in such scenarios moving the directories under non tmpfs and re-running ambari-server setup is the best option.

avatar
<br>

@Jay thank you so much for the solution

just one question - can we be sure that set the folders to be under /var and not under /var/run , not

Damage to ambari cluster functionality ?


first I will change the code in the ambari Kit in order to update ambari.properties file per your great recommendation

so bootstrap.dir , pid.dir , recommendations.dir will be under /var and not under /var/run (this is for scratch installation)

# grep run /etc/ambari-server/conf/ambari.properties

bootstrap.dir=/var/ambari-server/bootstrap
pid.dir=/var/ambari-server
recommendations.dir=/var/ambari-server/stack-recommendations<br>

in order to fix the current ambari cluster we will change ambari.properties the same as below ( scratch installation ) , and we will run ambari setup to recreate the missing folders

please just approve my steps

Michael-Bronson

avatar
Master Mentor

@Michael Bronson

Yes, your steps looks correct:

0. Stop ambari-server

# ambari-server stop


1. Change the PATH for those 3 properties mentioned in the ambari.properties file.

pid.dir=/var/ambari-server
bootstrap.dir=/var/ambari-server/bootstrap
recommendations.dir=/var/ambari-server/stack-recommendations<br>

2. Run the "ambari-server setup" and during the ambari-server setup process, when prompted to Customize user account for ambari-server daemon?, choose y
(Please do not change any other properties)

NOTE: Just to avoid any mistakes it is always better to take ambari DB dump (just to make sure that by mistake we do not make any other db changes)

3. Then restart ambari-server.

# ambari-server start

.

avatar

@Dear Jay

We performed the following "procedure" per you recommendation

Seems that procedure completed successfully but still stack-recommendations folder not created under /var/ambari-server

where we are wrong?

we change the following in ambari.properties before the procedure :

  1. pid.dir=/var/run/ambari-server
  2. bootstrap.dir=/var/run/ambari-server/bootstrap
  3. recommendations.dir=/var/run/ambari-server/stack-recommendations

The procedure:

NOTE - you mentioned that - "when prompted to Customize user account for ambari-server daemon?, choose y"

but this question not appears during the process

<br>[root@master02 ~]# ambari-server stop
Using python  /usr/bin/python
Stopping ambari-server
Waiting for server stop...
Ambari Server stopped


[root@master02 ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Ambari-server daemon is configured to run under user 'ambari'. Change this setting [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)?
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)?
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.


[root@master02 ~]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................................
Server started listening on 8080
DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
[root@master02 ~]# ls -ltr /var/ambari-server/bootstrap/
total 0


[root@master02 ~]# ls -ltr /var/ambari-server/
total 4

drwxr-xr-x 2 ambari root 6 Mar 12 14:10 bootstrap
-rw-r--r-- 1 root   root 6 Mar 12 14:10 ambari-server.pid


[root@master02 ~]# find /var -name  stack-recommendations
[root@master02 ~]#
Michael-Bronson

avatar
Master Mentor

@Michael Bronson

Please try making any small config change from ambari UI and then see that the dir "stack-recommendations" created or not? If there is nothing to recommend then you won't see the dir created.

avatar

@Jay , ok but do you have any API suggestion to perform some refresh or reload in order to create the folder , because we are working on production clusters and we not want to change something that not necessary

Michael-Bronson