Ambari-server failed to start after system reboot: Below is the error message i am getting. Any help is appreciated.

[#1]

Using python /usr/bin/python Ambari-server status Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 36, in <module> from ambari_server.dbConfiguration import DATABASE_NAMES, LINUX_DBMS_KEYS_LIST File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration.py", line 28, in <module> from ambari_server.serverConfiguration import decrypt_password_for_alias, get_ambari_properties, get_is_secure, \ File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 587, in <module> configDefaults = ServerConfigDefaults() File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 500, in __init__ super(ServerConfigDefaultsLinux, self).__init__() File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 384, in __init__ self.check_if_directories_writable([self.OUT_DIR, self.PID_DIR]) File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 436, in check_if_directories_writable raise FatalException(-1, "Unable to access {0} directory. Confirm the directory is created and is writable by Ambari Server user account '{1}'".format(directory, getpass.getuser())) ambari_commons.exceptions.FatalException: "Fatal exception: Unable to access directory. Confirm the directory is created and is writable by Ambari Server user account 'hdfs', exit code -1"

[#2]

@Manoj Nirale

Looks like you are trying to run the ambari server as "hdfs" user. Is that intended? Before system restart were you running ambari-server as "root" user or "hdfs" user?

Unable to access directory. Confirm the directory is created and is writable by Ambari Server user account 'hdfs', exit code -1"

.

The error indicates that most probably the user who is trying to run ambari-server does not have write permissions the PID file and the below mentioned directory.

ls -lart /var/run/ambari-server/ambari-server.pid
ls -ld /var/run/ambari-server

Please check the permissions or the user who is running the ambari.

.

[#3]

Thank you so much for your quick reply. Before system restarted it has 'root' user.

I even tried running with root user. But the same error.

Is there any problem with python version. My python version shows 2.7.5 but ambari-server config files are there in python 2.6 version.

[#4]

@Manoj Nirale

Python 2.6 and 2.7 both the versions are supported. Please see: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_support-matrices/content/ch_matrices-amb...

Python
For SLES 11:Python 2.6.x
For SLES 12:Python 2.7.x
For CentOS 7, Ubuntu 14, Ubuntu 16, and Debian7:Python 2.7.x

.

- What is your ambari version And Operating System Version?

- Permission error can not come with root user. Can you please share the latest error trace as earlier with "root" attempting to start the Ambari Server?

Following directory permission should be as following for root user:

# ls -lart /var/run/ambari-server/ambari-server.pid
-rw-r--r--. 1 root root 6 Jun  2 05:34 /var/run/ambari-server/ambari-server.pid

# ls -ld /var/run/ambari-server
drwxr-xr-x. 4 root root 4096 Jun  2 05:34 /var/run/ambari-server

.

[#5]

I could see there is no such folder called ambari-server.

Earlier there was no problem as such. only after the system restarted i'am facing this issue. please assist.

[root@node1 ~]# ls -lart /var/run/ambari-server/ambari-server.pid
ls: cannot access /var/run/ambari-server/ambari-server.pid: No such file or directory

[root@node1 ~]# ambari-server status
Using python  /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid
[root@node1 ~]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
ERROR: Exiting with exit code 1. 
REASON: Unable to detect a system user for Ambari Server.
- If this is a new setup, then run the "ambari-server setup" command to create the user
- If this is an upgrade of an existing setup, run the "ambari-server upgrade" command.

[#6]

@Manoj Nirale

Looks like permission issue. Can you try the following to reset the permissions: 1). Stop Ambari Server.

# ambari-server stop

2). Take ambari server Database Dump backup. (following is an example if you are using portgres for ambari DB). You will need to use your own credentials for ambari DB.

# pg_dump -Uambari > ambari_erie_9_june_2017.sql
Password:  bigdata

3). Now run the "ambari-server setup again" and make sure to choose "n" (NO) for almost all the options. Because we do not want to change anything.

# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y

Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? n
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Connecting to local database...done.
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Extracting system views...
...............

.

***** Remember to choose the following options: **** (NO for the following options)

Customize user account for ambari-server daemon [y/n] (n)? n
Do you want to change Oracle JDK [y/n] (n)? n
Enter advanced database configuration [y/n] (n)? n

4). Start ambari-server again

# ambari-server start

.

[#7]

Brother i am getting below error. I tried as per your answer. But could't able to resolve. kindly assist.

 Device or resource busy: '/root'

'

[root@node1 ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
ERROR: Failed: [Errno 16] Device or resource busy: '/root'
ERROR: Exiting with exit code 4. 
REASON: Failed to create user. Exiting.

[#8]

@Manoj Nirale

This looks like more of a Operating System Storage (Mount or disk) issue.

Device or resource busy: '/root

.

Please check if the Operating System Restart fixes the issue. Looks like because the device is busy it is now allowing new files to be created.