Zookeeper installation error

[#1]

Hi to all!

I am installing Zookeeper using Cloudera Manager 5.1.2 on Red Hat Enterprise Linux Server release 6.4 (Santiago) and I am keep getting this error message when installing it.

Completed only 0/1 steps. First failure: Command (744) has failed

Supervisor returned FATAL. Please check the role log file, stderr, or stdout.

Program: zookeeper/zkserver.sh ["1","/var/lib/zookeeper"]

Sat Sep 20 11:21:19 CEST 2014
+ source_parcel_environment
+ '[' '!' -z /pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh ']'
+ OLD_IFS='
'
+ IFS=:
+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)
+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)
+ IFS='
'
+ COUNT=1
++ seq 1 1
+ for i in '`seq 1 $COUNT`'
+ SCRIPT=/pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh
+ PARCEL_DIRNAME=CDH-5.1.2-1.cdh5.1.2.p0.3
+ . /pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh
/opt/cloudera-manager/cm-5.1.2/lib64/cmf/service/common/cloudera-config.sh: line 172: /pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh: Permission denied
+ locate_cdh_java_home
+ '[' -z /usr/java/jdk1.7.0_55 ']'
+ verify_java_home
+ '[' -z /usr/java/jdk1.7.0_55 ']'
+ echo JAVA_HOME=/usr/java/jdk1.7.0_55
+ '[' /usr/java/jdk1.7.0_55 '!=' '' ']'
+ JAVA=/usr/java/jdk1.7.0_55/bin/java
+ MYID=1
+ DATA_DIR=/var/lib/zookeeper
+ export ZOOKEEPER_HOME=/usr/lib/zookeeper
+ ZOOKEEPER_HOME=/usr/lib/zookeeper
+ echo 'using /usr/java/jdk1.7.0_55 as JAVA_HOME'
+ echo 'CDH_VERSION is 5'
+ echo 'ZOOKEEPER_HOME is /usr/lib/zookeeper'
+ echo 'configuration directory is /opt/cloudera-manager/cm-5.1.2/run/cloudera-scm-agent/process/51-zookeeper-server'
+ echo 'server id (myid) is 1'
+ echo 'data directory is /var/lib/zookeeper'
+ set_zookeeper_classpath
+ set_classpath_in_var ZOOKEEPER_CLASSPATH
+ '[' -z ZOOKEEPER_CLASSPATH ']'
+ '[' /opt/cloudera-manager/cm-5.1.2/share/cmf ']'
+ local INCLUDE_CDH_DIR=
+ '[' 5 ']'
+ INCLUDE_CDH_DIR=/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/cdh5
++ find /opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins /opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/cdh5 -maxdepth 1 -name '*.jar'
++ tr '\n' :
+ ADD_TO_CP=/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/event-publish-5.1.2-shaded.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/tt-instrumentation-5.1.2.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/cdh5/audit-plugin-cdh5-2.0.1-shaded.jar:
+ ADD_TO_CP=/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/event-publish-5.1.2-shaded.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/tt-instrumentation-5.1.2.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/cdh5/audit-plugin-cdh5-2.0.1-shaded.jar
+ eval 'OLD_VALUE=$ZOOKEEPER_CLASSPATH'
++ OLD_VALUE=
+ '[' -z ']'
+ export ZOOKEEPER_CLASSPATH=/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/event-publish-5.1.2-shaded.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/tt-instrumentation-5.1.2.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/cdh5/audit-plugin-cdh5-2.0.1-shaded.jar
+ ZOOKEEPER_CLASSPATH=/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/event-publish-5.1.2-shaded.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/tt-instrumentation-5.1.2.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/cdh5/audit-plugin-cdh5-2.0.1-shaded.jar
+ '[' '!' -d /var/lib/zookeeper ']'
+ echo 1
+ '[' '' '!=' '' ']'
+ '[' '' '!=' '' ']'
+ export 'ZOOKEEPER_SERVER_OPTS=-Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-mob1l0r0k.appdb.ngIBMD.prod.bide.de.tmo.log -Dzookeeper.log.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dzookeeper.datadir.autocreate=false -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/opt/cloudera-manager/cm-5.1.2/lib64/cmf/service/common/killparent.sh'
+ ZOOKEEPER_SERVER_OPTS='-Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-mob1l0r0k.appdb.ngIBMD.prod.bide.de.tmo.log -Dzookeeper.log.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dzookeeper.datadir.autocreate=false -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/opt/cloudera-manager/cm-5.1.2/lib64/cmf/service/common/killparent.sh'
+ exec /usr/java/jdk1.7.0_55/bin/java -cp '/opt/cloudera-manager/cm-5.1.2/run/cloudera-scm-agent/process/51-zookeeper-server:/usr/lib/zookeeper/lib/log4j.jar:/usr/lib/zookeeper/build/*:/usr/lib/zookeeper/build/lib/*:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/*:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/event-publish-5.1.2-shaded.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/tt-instrumentation-5.1.2.jar:/opt/cloudera-manager/cm-5.1.2/share/cmf/lib/plugins/cdh5/audit-plugin-cdh5-2.0.1-shaded.jar' -Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-mob1l0r0k.appdb.ngIBMD.prod.bide.de.tmo.log -Dzookeeper.log.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dzookeeper.datadir.autocreate=false -Xms1073741824 -Xmx1073741824 -XX:OnOutOfMemoryError=/opt/cloudera-manager/cm-5.1.2/lib64/cmf/service/common/killparent.sh org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/cloudera-manager/cm-5.1.2/run/cloudera-scm-agent/process/51-zookeeper-server/zoo.cfg
Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

It is the first service I started with to install on my (only) host.

I am using parcel CDH-5.1.2-1.cdh5.1.2.p0.3-el6.parcel and I installed CM5 with Installation Path C - Manual Installation Using Cloudera Manager Tarballs (I do not have internet access on the host where I am installing Cloudera Express)

I ran Host Inspector and all checks ran successfully.

Python version: Python 2.6.6

I am using external MySQL database:

mysql Ver 14.14 Distrib 5.6.20, for Linux (x86_64) using EditLine wrapper

Java version:

java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

I tried everything and I don't know what to do. Thanks for any help!

[#2]

Are you able to provide us the logs from the ZooKeeper instance
(/var/log/zookeeper)? It should tell us why it's not starting. Please paste
the logs into pastebin and provide the URL here. You just need to provide
the section covering the last startup attempt and the failure

[#3]

Thank you for your reply!

That folder is empty. Where can I set the log on?

Maybe CM can't find the files, libs and so on, since the parcel-repo folder is set to non-standard path for CM (not /opt/cloudera/parcel-repo). But on the other hand this should be okay since parcel for CDH itself can be seen by CM. I really don't know.

[#4]

The error message here might hold the key. Can you verify why it might not
be executable? Did you change permissions at some point?
​

/opt/cloudera-manager/cm-5.1.2/lib64/cmf/service/common/cloudera-config.sh:
line 172:
/pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh:
Permission denied

[#5]

That might be it.

I probably changed the owner of /opt/cloudera-manager/* directories and files from default user to "cloudera-scm". What owner should I re-set? I probably did it because of this error message you also pointed out.

For /pkg/moip/mo10755/work/mzpl/cloudera/parcels/* the owner is "root",

for /pkg/moip/mo10755/work/mzpl/cloudera/parcel-repo/* it is "cloudera-scm",

for /pkg/moip/mo10755/work/mzpl/cloudera/parcel-cache/* it is "root".

Thank you for your help!

[#6]

All those parcel directories should be manageable by Cloudera Manager,
should be owned by cloudera-scm:cloudera-scm

[#7]

I changed the owner of all parcels folder as you said, I restarted cloudera manager server and agent too, I also changed permissions of /opt/cloudera-manager/cm-5.1.2/lib64/cmf/service/common/cloudera-config.sh:

and 

/pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh

to 766 and I am still getting the same error.

[#8]

Did you try executing the script standalone, does it work?

/pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh

[#9]

I did and it doesn't work. That is strange.

-bash: /pkg/moip/mo10755/work/mzpl/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/meta/cdh_env.sh: Permission denied

[#10]

Can be here the problem? 

cat /etc/passwd

cloudera-scm:x:1551:1551:Cloudera SCM User;CHM02357619:/opt/cloudera-manager/cm-5.1.2/run/cloudera-scm-server:/bin/bash

I had to let our department create this user, so I am not sure if they really did this command:

sudo useradd --system --home=/opt/cloudera-manager/cm-5.1.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

Can you help me how to change it if this could be a problem? I don't want to harm the actual installation or host.

Thank you!

[#11 · Accepted solution]

I solved the problem.

cloudera-scm user did not have access to a parent folder: /pkg/moip/mo10755/work/mzpl which caused that the cloudera-scm user could not run any scripts from parcel subfolders.

I changed the permissions and now it works 🙂

Thank you very much for your help!

[#12]

Is /pkg/moip/mo10755/work/mzpl/cloudera/parcels or any of the directories
in between on a remote host or a separate mount? Do any of the mounts have
a noexec option? Run the mount command and see the flags set

​Check the #! line in the script, what shell does it refer to? Is that
program present on your box?

What user are you running the test as? 766 implies "other" cannot execute
the script.

To verify if the cloudera-scm user is valid, run the /bin/id command on it
(as root)
# /bin/id cloudera-scm​
​
Does the uid and gid match that in /etc/passwd?​