As a Hortonworks partner we encountered a problem after a clean install as described in previous version of tutorial how to install ambari we keep getting problems after rebooting the master server. The problem not only occurs on one distribution but on multiple (both CentOS 7 en Ubuntu 14.04 have been tested)
Essence of the problem is that folder /var/run/zeppelin is being removed on reboot of the master. If this folder is not there after reboot it has to be recreated and the correct permissions have to be addressed (mkdir -p /var/run/zeppelin && chown zeppelin:hadoop /var/run/zeppelin) prior to start the services in ambari.
If I don't do this the Service Zeppelin notebook will not start resulting in error below
resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/zeppelin-server/bin/zeppelin-daemon.sh restart >> /var/log/zeppelin/zeppelin-setup.log' returned 1. mkdir: cannot create directory ?/var/run/zeppelin?: Permission denied /usr/hdp/current/zeppelin-server/bin/zeppelin-daemon.sh: line 187: /var/run/zeppelin/zeppelin-zeppelin-master1.pid: No such file or directory cat: /var/run/zeppelin/zeppelin-zeppelin-master1.pid: No such file or directory
After that even after creation of that folder i cannot start Zeppelin notebook service because port is already in use. With ubuntu i can find the phantom process that is using that port with command netstat -ap |grep 9995 (assuming the zeppelin notebook port is 9995 as it is by default) and kill the process using it.
so it is a simple workaround: create folder /var/run/zeppelin and assign correct credentials
(mkdir -p /var/run/zeppelin && chown zeppelin:hadoop /var/run/zeppelin)
Question remains why is the folder deleted
See where /var/run is mounted: https://community.hortonworks.com/questions/48341/zeppelin-060-default-pid-directory.html It could be in memory.