Zookeeper Server keeps stopping after I restart with Ambar

[#1]

I'm running hdp 2.3.6 on AWS.

I restart zookeeper server & it starts without any error but quickly stops. The stderr has no entries.

I'd appreciate any help to debug this issues.

Thanks in advance,

Sai Dileep.

[#2]

@Sai Dileep

Please try the following:

1). Make sure that your "Advanced zookeeper-log4j" from Ambari UI has the following lines to make sure that we are running the zookeeper in DEBUG mode and the log file location is including the correct "${zookeeper.log.dir}"

log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE
.
.
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/zookeeper.log

2). Restart the zookeeper and then check the log. Please share the log.

/var/log/zookeeper/zookeeper.log 

3). Also please check the OS resources are enough or not?

# free  -m
# top

4). Try deleting the PID file of zookeerper and then restart again.

/var/run/zookeeper/zookeeper_server.pid 

.

[#3]

@Jay SenSharma

Thanks for the quick response.

I went through the logs and found that some other process is running on the port 2181.

I killed the process and restarted zookeeper. Now it is running fine.

Btw, My log file is named zookeeper-machine ip -ec2.out.

I modified the log4j.appender.ROLLINGFILE.File value as /var/log/zookeeper/zookeeper.log. Still, it does'nt rename.

Thanks,

Sai Dileep.

[#4]

@Sai Dileep

Good to know that your original issue is resolved.

For the logging related issue please check what is the output of the following command and if your changes are getting reflected there? Specially the "zookeeper.log.file" and the "zookeeper.root.logger" if it is pointing correctly

# ps -ef | grep zookeeper
516        969     0  0 Apr03 ?        00:01:52 /usr/lib/jvm/java/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.log.file=zookeeper-zookeeper-server-sandbox.hortonworks.com.log -Dzookeeper.root.logger=INFO,ROLLINGFILE   ........................

.

Although it is good practice to have the hostname appended to the zookeeper log file name that way it makes it easier for us to differentiate between logs from different hosts.

# grep 'ZOO_LOG_FILE' /usr/hdp/current/zookeeper-server/bin/zkServer.sh 
ZOO_LOG_FILE=zookeeper-$USER-server-$HOSTNAME.log

.

[#5]

I have the same problem; and I do not have the file: zookeeper.log!!

[#6]

@Maryem Mary

Your log4j property wasn't configured suitably.

Ensure all highlighted properties are updated in log4j.properties as follow. "Ambari -> Zookeeper -> Configs -> Advanced zookeeper-log4j"

#
# ZooKeeper Logging Configuration
#
# DEFAULT: console appender only

# log4j.rootLogger=INFO, CONSOLE

zookeeper.root.logger=INFO,console
zookeeper.log.dir=.
zookeeper.log.file=zookeeper.log

log4j.rootLogger=${zookeeper.root.logger}

...

...

#
# Add ROLLINGFILE to rootLogger to get log file output
# Log DEBUG level and above messages to a log file
log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=DEBUG
# log4j.appender.ROLLINGFILE.File=zookeeper.log
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file}

# Max log file size of 256MB
log4j.appender.ROLLINGFILE.MaxFileSize=256MB
# uncomment the next line to limit number of backup files
log4j.appender.ROLLINGFILE.MaxBackupIndex=10

..

..

[#7]

I did what you said what the same problem zookeeper keeps to stops every 10 s. This the content of

log4j.properties:

# ZooKeeper Logging Configuration
# DEFAULT: console appender only
log4j.rootLogger=INFO, CONSOLE, ROLLINGFILE

zookeeper.log.dir=.
zookeeper.log.file=zookeeper.log

log4j.rootLogger=${zookeeper.root.logger}

# Example with rolling log file
#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE

# Example with rolling log file and tracing
#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE

#
# Log INFO level and above messages to the console
#
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=INFO
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n

#
# Add ROLLINGFILE to rootLogger to get log file output
# Log DEBUG level and above messages to a log file
log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=DEBUG
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file}

# Max log file size of 10MB
log4j.appender.ROLLINGFILE.MaxFileSize={{zookeeper_log_max_backup_size}}MB
# uncomment the next line to limit number of backup files
log4j.appender.ROLLINGFILE.MaxBackupIndex=10

[#8]

@Maryem Mary Share the zookeeper log or error trace to find a cause, as above configuration for enabling the logs.

[#9]

There is no log file created in ZooKeeper Log Dir :/var/log/zookeeper

[#10]

@Sai Dileep

The only way to have a look at these types of error is directly checking them in the log file (As shown by @Jay Kumar SenSharma already) or you can also take a different route by using Ambari log search service. Incase you don't have it in your environment it's a good idea to install it. But it's recommended only if you have a cluster size comprising of 150 or lesser nodes.

Also, this type of behaviour is normal when a port is already binded by another process Zookeeper tries to start a process on the same port. Moreover, in ambari console like the image you have provided above it is not reflected reason being it captures the log till the pid is started and not after that.

Here is the gify showing same type of symptoms with history server.output-sy8pf5.gif