Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can't access my Ambari Dashboard after the system shutdown abnormally

Can't access my Ambari Dashboard after the system shutdown abnormally

New Contributor

@abajwa @TimothySpann @KuldeepK 

I have a master node and other 3 nodes, everything was working fine, suddenly there was power interruption and all the 4 nodes went down and on restarting I can't my dashboard using the IP Address. (This site can’t be reached

192.168.4.15 refused to connect.

 

ERR_CONNECTION_REFUSED)
 

I make use of this command "[root@masternode ~]# ambari-server reset" and below is the output i got.

[root@masternode ~]# ambari-server reset
Using python /usr/bin/python
Resetting ambari-server
ERROR: Could not create . Reason: [Errno 2] No such file or directory: ''
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 37, in <module>
from ambari_server.dbConfiguration import DATABASE_NAMES, LINUX_DBMS_KEYS_LIST
File "/usr/lib/ambari-server/lib/ambari_server/dbConfiguration.py", line 30, in <module>
from ambari_server.serverConfiguration import decrypt_password_for_alias, get_ambari_properties, get_is_secure, \
File "/usr/lib/ambari-server/lib/ambari_server/serverConfiguration.py", line 603, in <module>
configDefaults = ServerConfigDefaults()
File "/usr/lib/ambari-server/lib/ambari_server/serverConfiguration.py", line 512, in __init__
super(ServerConfigDefaultsLinux, self).__init__()
File "/usr/lib/ambari-server/lib/ambari_server/serverConfiguration.py", line 396, in __init__
self.check_if_directories_writable([self.OUT_DIR, self.PID_DIR])
File "/usr/lib/ambari-server/lib/ambari_server/serverConfiguration.py", line 448, in check_if_directories_writable
raise FatalException(-1, "Unable to access {0} directory. Confirm the directory is created and is writable by Ambari Server user account '{1}'".format(directory, getpass.getuser()))
ambari_commons.exceptions.FatalException: "Fatal exception: Unable to access directory. Confirm the directory is created and is writable by Ambari Server user account 'root', exit code -1"
 

 

I checked this "check /etc/ambari-server/conf/ambari.properties" and it's empty

 

Thanks

4 REPLIES 4

Re: Can't access my Ambari Dashboard after the system shutdown abnormally

New Contributor

someone should please help me out

Highlighted

Re: Can't access my Ambari Dashboard after the system shutdown abnormally

Super Collaborator

Assuming that PID directory is /var/run(or /run) which is usually a tmpfs will have owned by root user and all the pid directories will be lost during server reboots. 

 

As ambari-server has its PID under /var/run, after restart of server, it tries to create directory /run/ambari-server/ and fails if ambari server is running as non-root user. 

 

Make sure /var/run/ambari-server/ is created on ambari host and has correct ownership  to allow write access to non-root user. 

 

Also verify /var/log/ambari-server dir permissions/ownership and try starting service again. 

 

Note that ambari-server reset will wipe out all the ambari DB so you have to create cluster from scratch again. 

Re: Can't access my Ambari Dashboard after the system shutdown abnormally

Super Collaborator

From error, I see it is root user. so please verify the mount directories are mounted with rw option. 

#mount -v 

Re: Can't access my Ambari Dashboard after the system shutdown abnormally

Mentor

@DareBukky 

 

It's usually recommended to do search the community forum before running such a command especially if we are not sure of what it does there are tons of info from the merged HCC and Cloudera community,.

In your case the reset ambari server deletes all the cluster data in Ambari server database and replaces it with the default schema. It does not touch any namenode data/binaries.

That explains why your ambari.properties is empty !!

First step backup all the db's

For MySQL:
mysqldump -u ambari –p bigdata ambari > /tmp/db_backup.sql

For PostgreSQL:
pg_dump -U ambari -f /tmp/db_backup.sql

By precaution do the above for all components hive, ranger,  oozie, druid, Rangerkms etc

 

Can you check the backend db's for Ambari,hive,ranger etc Assuming you are on Mysql/MariaDB

mysql -u root -p

mysql> show databases;

If the [Ambari database exist] then try logging as ambari user to that database below I am assuming user and and password is Ambari

mysql -u ambari -pambari
mysql> use ambari; [if that was the ambari db before the reset]

Validate that the database is healthy!

Mysql [ambari]> show tables;

You should have [111 rows in set (0.00 sec)]

Depending on the outcome of the above steps then there is a path to follow.
Please revert

Don't have an account?
Coming from Hortonworks? Activate your account here