Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Zookeeper Server keeps stopping after I restart with Ambar

Highlighted

Zookeeper Server keeps stopping after I restart with Ambar

I'm running hdp 2.3.6 on AWS.

I restart zookeeper server & it starts without any error but quickly stops. The stderr has no entries.

I'd appreciate any help to debug this issues.

Thanks in advance,

Sai Dileep.

16 REPLIES 16

Re: Zookeeper Server keeps stopping after I restart with Ambar

Super Mentor

@Sai Dileep

Please try the following:

1). Make sure that your "Advanced zookeeper-log4j" from Ambari UI has the following lines to make sure that we are running the zookeeper in DEBUG mode and the log file location is including the correct "${zookeeper.log.dir}"

log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE
.
.
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/zookeeper.log

2). Restart the zookeeper and then check the log. Please share the log.

/var/log/zookeeper/zookeeper.log 

3). Also please check the OS resources are enough or not?

# free  -m
# top

4). Try deleting the PID file of zookeerper and then restart again.

/var/run/zookeeper/zookeeper_server.pid 

.

Re: Zookeeper Server keeps stopping after I restart with Ambar

@Jay SenSharma

Thanks for the quick response.

I went through the logs and found that some other process is running on the port 2181.

I killed the process and restarted zookeeper. Now it is running fine.

Btw, My log file is named zookeeper-machine ip -ec2.out.

I modified the log4j.appender.ROLLINGFILE.File value as /var/log/zookeeper/zookeeper.log. Still, it does'nt rename.

Thanks,

Sai Dileep.

Re: Zookeeper Server keeps stopping after I restart with Ambar

Super Mentor

@Sai Dileep

Good to know that your original issue is resolved.

For the logging related issue please check what is the output of the following command and if your changes are getting reflected there? Specially the "zookeeper.log.file" and the "zookeeper.root.logger" if it is pointing correctly

# ps -ef | grep zookeeper
516        969     0  0 Apr03 ?        00:01:52 /usr/lib/jvm/java/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.log.file=zookeeper-zookeeper-server-sandbox.hortonworks.com.log -Dzookeeper.root.logger=INFO,ROLLINGFILE   ........................

.

Although it is good practice to have the hostname appended to the zookeeper log file name that way it makes it easier for us to differentiate between logs from different hosts.

# grep 'ZOO_LOG_FILE' /usr/hdp/current/zookeeper-server/bin/zkServer.sh 
ZOO_LOG_FILE=zookeeper-$USER-server-$HOSTNAME.log

.

Re: Zookeeper Server keeps stopping after I restart with Ambar

Explorer

I have the same problem; and I do not have the file: zookeeper.log!!

Re: Zookeeper Server keeps stopping after I restart with Ambar

Contributor

@Maryem Mary

Your log4j property wasn't configured suitably.

Ensure all highlighted properties are updated in log4j.properties as follow. "Ambari -> Zookeeper -> Configs -> Advanced zookeeper-log4j"

#
# ZooKeeper Logging Configuration
#
# DEFAULT: console appender only

# log4j.rootLogger=INFO, CONSOLE


zookeeper.root.logger=INFO,console
zookeeper.log.dir=.
zookeeper.log.file=zookeeper.log

log4j.rootLogger=${zookeeper.root.logger}

...

...

#
# Add ROLLINGFILE to rootLogger to get log file output
# Log DEBUG level and above messages to a log file
log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=DEBUG
# log4j.appender.ROLLINGFILE.File=zookeeper.log
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file}


# Max log file size of 256MB
log4j.appender.ROLLINGFILE.MaxFileSize=256MB
# uncomment the next line to limit number of backup files
log4j.appender.ROLLINGFILE.MaxBackupIndex=10

..

..

Re: Zookeeper Server keeps stopping after I restart with Ambar

Explorer

I did what you said what the same problem zookeeper keeps to stops every 10 s. This the content of

log4j.properties:

# ZooKeeper Logging Configuration
# DEFAULT: console appender only
log4j.rootLogger=INFO, CONSOLE, ROLLINGFILE

zookeeper.log.dir=.
zookeeper.log.file=zookeeper.log

log4j.rootLogger=${zookeeper.root.logger}

# Example with rolling log file
#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE

# Example with rolling log file and tracing
#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE

#
# Log INFO level and above messages to the console
#
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=INFO
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n

#
# Add ROLLINGFILE to rootLogger to get log file output
# Log DEBUG level and above messages to a log file
log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=DEBUG
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file}

# Max log file size of 10MB
log4j.appender.ROLLINGFILE.MaxFileSize={{zookeeper_log_max_backup_size}}MB
# uncomment the next line to limit number of backup files
log4j.appender.ROLLINGFILE.MaxBackupIndex=10

Re: Zookeeper Server keeps stopping after I restart with Ambar

Contributor

@Maryem Mary Share the zookeeper log or error trace to find a cause, as above configuration for enabling the logs.

Re: Zookeeper Server keeps stopping after I restart with Ambar

Explorer

There is no log file created in ZooKeeper Log Dir :/var/log/zookeeper

Re: Zookeeper Server keeps stopping after I restart with Ambar

Contributor

@Sai Dileep

The only way to have a look at these types of error is directly checking them in the log file (As shown by @Jay Kumar SenSharma already) or you can also take a different route by using Ambari log search service. Incase you don't have it in your environment it's a good idea to install it. But it's recommended only if you have a cluster size comprising of 150 or lesser nodes.

Also, this type of behaviour is normal when a port is already binded by another process Zookeeper tries to start a process on the same port. Moreover, in ambari console like the image you have provided above it is not reflected reason being it captures the log till the pid is started and not after that.

Here is the gify showing same type of symptoms with history server.output-sy8pf5.gif

Don't have an account?
Coming from Hortonworks? Activate your account here