Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cant start kafka broker

avatar

we have in our Hadoop cluster 3 kafka brokers ( based on ambari )

one of the kafka broker can’t starting ( kafka01 )

any suggestion for this situation ?

we have the following logs

from /var/log/kafka/server.log

from /var/log/kafka/server.log: FATAL Fatal error<br>
during KafkaServer shutdown. (kafka.server.KafkaServer)java.lang.IllegalStateException: Kafka server is still
starting up, cannot shut down!  at
kafka.server.KafkaServer.shutdown(KafkaServer.scala:576)  at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:51)  at
kafka.Kafka$$anon$1.run(Kafka.scala:63)[2018-05-10 14:23:57,032] FATAL Fatal error during
KafkaServerStable shutdown. Prepare to halt (kafka.server.KafkaServerStartable)java.lang.IllegalStateException: Kafka server is still
starting up, cannot shut down!  at
kafka.server.KafkaServer.shutdown(KafkaServer.scala:576)  at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:51)  at
kafka.Kafka$$anon$1.run(Kafka.scala:63)[2018-05-10 14:23:59,867] INFO KafkaConfig values:


from /var/log/kafka/kafka.err


Exception in thread "metrics-meter-tick-thread-3"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-2"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-5"
java.lang.OutOfMemoryError: Java heap spaceException: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"kafka-socket-acceptor-PLAINTEXT-6667"Exception in thread "metrics-meter-tick-thread-4"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-7"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-6"
java.lang.OutOfMemoryError: Java heap space

we not get any output from the following: ( port isn't licensing )

netstat -tnlpa | grep 6667

Michael-Bronson
6 REPLIES 6

avatar
Super Collaborator

The recommendation here would be to increase the heap space allocated to the Kafka process or reduce the amount of other processes running on the same server. For example, in a production environment, the Kafka brokers should be standalone servers -- not on the same hardware as Zookeeper or other Hadoop processes.

avatar

@Jordan , regarding to what you said - "increase the heap space allocated to the Kafka process" can you give example of the parameter? , so we can find it from ambari GUI , second yes this kafka broker is stand alone machine and not with the Zookeeper

Michael-Bronson

avatar

I am not sure about the following , but do you mean to update the file - /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh and update the parameter - export KAFKA_HEAP_OPTS="-Xms2G -Xmx2G" ? ( according to the article - https://community.hortonworks.com/content/supportkb/151841/error-javalangoutofmemoryerror-direct-buf... )

[root@kafka01 conf]# more /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ $# -lt 1 ];
then
        echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
        exit 1
fi
base_dir=$(dirname $0)
if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi
EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}
COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    shift
    ;;
  *)
    ;;
esac
echo $KAFKA_HEAP_OPTS>>/tmp/uri
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
Michael-Bronson

avatar

@Jordan , another quastion please

regarding the file - /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh , I see that the default value - is KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" , so my question is , based on both variables that set to 1G , is it logical that 1G isn't enough ?

Michael-Bronson

avatar

@Jordan , another point , I forget to tell you that we also restart the kafka machine but this not help to resolve the kafka broker , so the option is to increase the value from 1G to 2G according to your solution ,

Michael-Bronson

avatar
Super Collaborator

@Michael Bronson

Kafka stores the latest offsets in memory before they are sent to disk, therefore, the more memory the better, with a max of 8G.

And I would assume that the heap properties can be set from Ambari rather than individually on the broker, but I don't use Kafka from HDP, so I can't say.