Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Custom CSD starts with an error

Custom CSD starts with an error

Explorer

I wrote a CSD for Zeppelin and installed it. After adding the service in CM, it starts on the server itself but in the Cloudera Manager I get a message: Failed to start service.

 

Here is a more detailed error:

Wed Dec 20 12:41:21 CET 2017
JAVA_HOME=/usr/java/jdk1.8.0_92
Using /run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER
CMF_CONF_DIR=/etc/cloudera-scm-agent
Starting the zeppelin server
Zeppelin start [  OK  ]
Wed Dec 20 12:41:25 CET 2017
JAVA_HOME=/usr/java/jdk1.8.0_92
Using /run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER
CMF_CONF_DIR=/etc/cloudera-scm-agent
Starting the zeppelin server
Zeppelin is already running
Wed Dec 20 12:41:27 CET 2017
JAVA_HOME=/usr/java/jdk1.8.0_92
Using /run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER
CMF_CONF_DIR=/etc/cloudera-scm-agent
Starting the zeppelin server
Zeppelin start [  OK  ]
Wed Dec 20 12:41:32 CET 2017
JAVA_HOME=/usr/java/jdk1.8.0_92
Using /run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/run/cloudera-scm-agent/process/1225-zeppelin-ZEPPELIN_WEBSERVER
CMF_CONF_DIR=/etc/cloudera-scm-agent
Starting the zeppelin server
Zeppelin start [  OK  ]

 

It looks to me like it's trying to start the service multiple times and fails since it has already been started once.

 

Here is my service.sdl file:

{
  "name" : "ZEPPELIN",
  "label" : "Apache Zeppelin",
  "description" : "The Zeppelin Service Running on GW node",
  "version" : "0.7.3",
  "icon" : "images/icon.png",
  "runAs" : { 
    "user" : "zeppelin",
    "group" : "zeppelin"
   },  
   "roles" : [
    {
       "name" : "ZEPPELIN_WEBSERVER",
       "label" : "Zeppelin Server",
       "pluralLabel" : "Zeppelin Servers",
       "startRunner" : {
         "program" : "scripts/control.sh",
         "args" : [ "start" ]
       },
       "stopRunner" : {
         "program" : "scripts/control.sh",
         "args" : [ "stop" ]
       },
       "externalLink" : {
        "name" : "zeppelin_web_ui",
        "label" : "Zeppelin Webserver UI",
        "url" : "http://${host}:8090",
        "secureUrl" : "https://${host}:8443"
       },
       "logging" : {
        "dir" : "/var/log/zeppelin_service",
        "filename" : "zeppelin_service.INFO",
        "modifiable" : true,
        "loggingType" : "glog"
       }
    }
  ],
  "rolesWithExternalLinks" : ["ZEPPELIN_WEBSERVER"]
}

and control.sh:

#!/bin/bash
CMD=$1

case $CMD in
  (start)
    echo "Starting the zeppelin server"
    exec /app/zeppelin/0.7.3/bin/zeppelin-daemon.sh start
    ;;
  (stop)
    echo "Stopping the zeppelin server"
    exec /app/zeppelin/0.7.3/bin/zeppelin-daemon.sh stop
    ;;
  (*)
    echo "Don't understand [$CMD]"
    ;;
esac

Any ideas what could cause the problem? Thanks!

4 REPLIES 4

Re: Custom CSD starts with an error

Hi,

Does "zeppelin-daemon.sh start" start zeppelin as a foreground process, or
does it fork? You need to start the daemon as a foreground process so that
the PID that CM is monitoring is the same as the PID of the daemon. If it
forks, then CM is monitoring just the shell script, sees that the shell
script exits shortly after starting, thinks that is the daemon crashing
out, and retries starting it up to 2 more times.

Re: Custom CSD starts with an error

Explorer

As far as I can tell it forks. In the error message of CM I can see

 

Thu Dec 21 08:40:53 CET 2017
JAVA_HOME=/usr/java/jdk1.8.0_92
Using /run/cloudera-scm-agent/process/1250-zeppelin-ZEPPELIN_WEBSERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/run/cloudera-scm-agent/process/1250-zeppelin-ZEPPELIN_WEBSERVER
CMF_CONF_DIR=/etc/cloudera-scm-agent
Starting the zeppelin server
Zeppelin is already running

So, CM gives it a process of 1250 and on the server it's 36767:

zeppelin 36767     1 14 08:40 ?        00:00:08 /usr/java/jdk1.8.0_92/bin/java -Dfile.encoding=UTF-8 -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -Dlog4j.configuration=file:///app/zeppelin/0.7.3/conf/log4j.properties -Dzeppelin.log.file=/app/zeppelin/0.7.3/logs/zeppelin--XXXXXXXXXXX.log -cp ::/app/zeppelin/0.7.3/lib/interpreter/*:/app/zeppelin/0.7.3/lib/*:/app/zeppelin/0.7.3/*::/app/zeppelin/0.7.3/conf org.apache.zeppelin.server.ZeppelinServer

Is there a possibility to deal with this forking?

 

Re: Custom CSD starts with an error

New Contributor

@ahaeni wrote:

As far as I can tell it forks. In the error message of CM I can see

 

Thu Dec 21 08:40:53 CET 2017
JAVA_HOME=/usr/java/jdk1.8.0_92
Using /run/cloudera-scm-agent/process/1250-zeppelin-ZEPPELIN_WEBSERVER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/run/cloudera-scm-agent/process/1250-zeppelin-ZEPPELIN_WEBSERVER
CMF_CONF_DIR=/etc/cloudera-scm-agent
Starting the zeppelin server
Zeppelin is already running

So, CM gives it a process of 1250 and on the server it's 36767:

zeppelin 36767     1 14 08:40 ?        00:00:08 /usr/java/jdk1.8.0_92/bin/java -Dfile.encoding=UTF-8 -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -Dlog4j.configuration=file:///app/zeppelin/0.7.3/conf/log4j.properties -Dzeppelin.log.file=/app/zeppelin/0.7.3/logs/zeppelin--XXXXXXXXXXX.log -cp ::/app/zeppelin/0.7.3/lib/interpreter/*:/app/zeppelin/0.7.3/lib/*:/app/zeppelin/0.7.3/*::/app/zeppelin/0.7.3/conf org.apache.zeppelin.server.ZeppelinServer

Is there a possibility to deal with this forking?

 


Hello. 

 

Opening the script, you see that in the function start() there is this command

nohup nice -n $ZEPPELIN_NICENESS $ZEPPELIN_RUNNER $JAVA_OPTS -cp $ZEPPELIN_CLASSPATH_OVERRIDES:$CLASSPATH $ZEPPELIN_MAIN >> "${ZEPPELIN_OUTFILE}" 2>&1 < /dev/null &

 Try removing the "&" at the end and see if it works. 

 

If it does, be aware that this might break the rest of the script, so you should do additional checks and see how it affects it, and modify more accordingly.

Highlighted

Re: Custom CSD starts with an error

New Contributor
Actually, now that I wanted to do this exact thing myself (installing Zeppelin as CSD, when I initially replied, I wanted to install Azkaban).
The better solution was to just use "upstart" instead of "start"
Don't have an account?
Coming from Hortonworks? Activate your account here