Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

zeppelin 0.6.0 - default pid directory

Highlighted

zeppelin 0.6.0 - default pid directory

Expert Contributor

Im using Ubuntu 14.04. When adding service Zeppelin 0.6.0 via Ambari, the default pid directory is set to /var/run/zeppelin. The directory gets created when Zeppelin is installed. The problem arises when ubuntu instance gets rebooted and then the zeppelin directory gets deleted. When starting Zeppelin service after the reboot, the service wil not start because /var/run/zeppelin does not exist.

Solution: The way I solved is that I created a run directory under $ZEPPELIN_HOME and so the zeppelin pid directory was changed in Ambari to $ZEPPELIN_HOME/run.

Appreciate a confirmation from Hortonworks if this is best practice in this case.

9 REPLIES 9
Highlighted

Re: zeppelin 0.6.0 - default pid directory

Super Guru

@marko

/var/run is usually mounted as tmpfs, which is a partition mapped into your RAM. Obviously RAM gets cleared on reboot, so do all tmpfs file systems.

Can you please check if this is true in your case, in that case you might want to mount /var/run/zeppelin to some other disk and this might be a bug. Like other services, Zeppelin should create /var/run/zeppelin at service restart. Please check and file a bug if this is true.

Highlighted

Re: zeppelin 0.6.0 - default pid directory

Expert Contributor

@Kuldeep Kulkarni

/var/run is linked to /run which is mounted as tmpfs, yes. I believe this is standard Ubuntu configuration. Mounting /var/run/zeppelin to another volume sounds more work than changing the pid directory (which I did). I took out the previous Zeppelin config version to try it out again:

zeppelin_pid_dir = /var/run/zeppelin-notebook

Upon starting it I get this in the error window in ambari:

resource_management.core.exceptions.Fail: Execution of '/usr/hdp/current/zeppelin-server/lib/bin/zeppelin-daemon.sh start >> /var/log/zeppelin/zeppelin-setup.log' returned 1. mkdir: cannot create directory ‘/var/run/zeppelin-notebook’: Permission denied
/usr/hdp/current/zeppelin-server/lib/bin/zeppelin-daemon.sh: line 182: /var/run/zeppelin-notebook/zeppelin-zeppelin-t-client01.pid: No such file or directory
cat: /var/run/zeppelin-notebook/zeppelin-zeppelin-t-client01.pid: No such file or directory

Changing zeppelin_pid_dir back to $ZEPPELIN_HOME/run and restarting delivers:

Execute['echo pid file is: /usr/hdp/2.4.0.0-169/zeppelin/run/zeppelin-zeppelin-t-client01.pid'] {'user': 'zeppelin'}
Highlighted

Re: zeppelin 0.6.0 - default pid directory

Super Guru

@marko

Can you please check permissions of parent directory(/var/run)? Or did you create /var/run/zeppelin-notebook manually by root user? In that case I belive zeppelin user wont be able to write into it

I see below error

mkdir: cannot create directory ‘/var/run/zeppelin-notebook’:Permission denied
Highlighted

Re: zeppelin 0.6.0 - default pid directory

Expert Contributor
@Kuldeep Kulkarni
ll /var/run

Returns the following:

lrwxrwxrwx 1 root root 4 Apr 11 22:40 /var/run -> /run/

I havent created anything manually, no.

This worked when I installed the Zeppelin service, but after restarting the instance and restarting the Zeppelin service, the problem occured.

And by the way, is it maybe better to name both zeppelin directories in /var/run and /var/log the same? Or is there a reason for having zeppelin in logs and zeppelin-notebook in run?

Highlighted

Re: zeppelin 0.6.0 - default pid directory

Super Guru

this is symlink which will always show lrwxrwxrwx, can you please do ls -lrtd /run and paste the output?

yeah for logs it is standard directory structure /var/log/<service>

Highlighted

Re: zeppelin 0.6.0 - default pid directory

Super Guru
Highlighted

Re: zeppelin 0.6.0 - default pid directory

Expert Contributor

@Kuldeep Kulkarni

Yes,

Heres the output:

drwxr-xr-x 27 root root 920 Aug  1 08:58 /run

All the other services work (restart) fine and I havent changed any permission for /run.

Highlighted

Re: zeppelin 0.6.0 - default pid directory

Super Guru

Thanks @marko

Maybe this is a bug, because of above permissions zeppelin user is unable to create directory inside /var/run

Can you please file a bug at https://issues.apache.org with re-producable steps?

Below command gets failed in your case

  if [[ ! -d "${ZEPPELIN_PID_DIR}" ]]; then
    echo "Pid dir doesn't exist, create ${ZEPPELIN_PID_DIR}"
    $(mkdir -p "${ZEPPELIN_PID_DIR}")
  fi

Re: zeppelin 0.6.0 - default pid directory

New Contributor

The issue here is, that I think Zeppelin is trying to create the pid dir as "zeppelin", while other services do this as root through Ambari.

I've run into an additional issue, where in Ambari the PID_FILE-setting is greyed out, and impossible to override, so the provided workaround won't work.

As a proper fix, I would suggest patching Zeppelin's Ambari integration, so that ambari will create the directory as root, then grant the proper rights to the zeppelin user who will manage the PID file.

Don't have an account?
Coming from Hortonworks? Activate your account here