Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to restart HDP sandbox after shutdown server?

How to restart HDP sandbox after shutdown server?

New Contributor

Hi, I'm new to HDP sandbox. I installed HDP 2.6.3 sandbox for docker in centos server.

After shutdown and restart server, I started sandbox-hdp container but Ambari and other service didn't start.

So I remote to container and manually started ambari-server, ambari-agent and postgresql. Then I login to Ambari UI and try to start other service but UI show that the process was aborted and service didn't start.

Please advice how to fix this. And if I stop or start service in the wrong way, please suggest how to do it.

Thank you in advance. (Sorry for my bad english)

Log when manually started service:

[root@sandbox-hdp ~]# service ambari-agent status
Found ambari-agent PID: 1390
ambari-agent not running. Stale PID File at: /var/run/ambari-agent/ambari-agent.pid
[root@sandbox-hdp ~]# service ambari-server status
Using python  /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid
[root@sandbox-hdp ~]# service postgresql status
postmaster dead but pid file exists
[root@sandbox-hdp ~]# ambari-agent restart
Restarting ambari-agent
Verifying Python version compatibility...
Using python  /usr/bin/python
Found ambari-agent PID: 1390
ERROR: ambari-agent not running. Stale PID File at: /var/run/ambari-agent/ambari-agent.pid
Removing PID file at /var/run/ambari-agent/ambari-agent.pid
ambari-agent successfully stopped
Verifying Python version compatibility...
Using python  /usr/bin/python
Checking for previously running Ambari Agent...
chown: cannot access `/var/lib/ambari-agent/data/command-10.json': No such file or directory
chown: cannot access `/var/lib/ambari-agent/data/command-102.json': No such file or directory
...
chown: cannot access `/var/lib/ambari-agent/data/errors-10.txt': No such file or directory
chown: cannot access `/var/lib/ambari-agent/data/errors-102.txt': No such file or directory
...
chown: cannot access `/var/lib/ambari-agent/data/output-10.txt': No such file or directory
chown: cannot access `/var/lib/ambari-agent/data/output-102.txt': No such file or directory
...
chown: cannot access `/var/lib/ambari-agent/data/structured-out-10.json': No such file or directory
chown: cannot access `/var/lib/ambari-agent/data/structured-out-102.json': No such file or directory
...
chown: cannot access `/var/lib/ambari-agent/data/structured-out-9.json': No such file or directory
Checking ambari-common dir...
Starting ambari-agent
Verifying ambari-agent process status...
Ambari Agent successfully started
Agent PID at: /var/run/ambari-agent/ambari-agent.pid
Agent out at: /var/log/ambari-agent/ambari-agent.out
Agent log at: /var/log/ambari-agent/ambari-agent.log
[root@sandbox-hdp ~]# service postgresql restart
Stopping postgresql service:                               [FAILED]
Starting postgresql service:                               [  OK  ]
[root@sandbox-hdp ~]# ambari-server restart
Using python  /usr/bin/python
Restarting ambari-server
Ambari Server is not running
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start..............................
Server started listening on 8080


DB configs consistency check: no errors and warnings were found.

Ambari screen:

https://photos.app.goo.gl/VHbDdCZRxvq5GuVB2

https://photos.app.goo.gl/y8J2zTW6ikPyKsb23

1 REPLY 1

Re: How to restart HDP sandbox after shutdown server?

Super Mentor

@atom siri

Looks like due to some reason the permission on the following directory is not setup correctly or there is some issue in the following directory file/creation:

chown: cannot access `/var/lib/ambari-agent/data/command-10.json': No such file or directory
chown: cannot access `/var/lib/ambari-agent/data/command-102.json': No such file or directory

.

So i will suggest you to please try creating the directory manually as following and then see if you are able to create a file there or not? (this is to isolate the issue)

[root@sandbox ~]# ls -ld /var/lib/ambari-agent
drwxr-xr-x 1 root root 4096 Sep 18 14:19 /var/lib/ambari-agent

[root@sandbox ~]# ls -ld /var/lib/ambari-agent/data
drwx------ 1 root root 118784 Jan 22 01:53 /var/lib/ambari-agent/data

.

After manually creating the directory please try creating a file on your own to see if you are facing any issue?

[root@sandbox ~]# echo "{}" > /var/lib/ambari-agent/data/command-10.json
[root@sandbox ~]# cat /var/lib/ambari-agent/data/command-10.json
{}

.

Sometimes restarting the VM fixes the filesystem issues. So better to try restarting the sandbox once.