Cant start kafka broker

[#1]

we have in our Hadoop cluster 3 kafka brokers ( based on ambari )

one of the kafka broker can’t starting ( kafka01 )

any suggestion for this situation ?

we have the following logs

from /var/log/kafka/server.log

from /var/log/kafka/server.log: FATAL Fatal error<br>

during KafkaServer shutdown. (kafka.server.KafkaServer)java.lang.IllegalStateException: Kafka server is still
starting up, cannot shut down!  at
kafka.server.KafkaServer.shutdown(KafkaServer.scala:576)  at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:51)  at
kafka.Kafka$$anon$1.run(Kafka.scala:63)[2018-05-10 14:23:57,032] FATAL Fatal error during
KafkaServerStable shutdown. Prepare to halt (kafka.server.KafkaServerStartable)java.lang.IllegalStateException: Kafka server is still
starting up, cannot shut down!  at
kafka.server.KafkaServer.shutdown(KafkaServer.scala:576)  at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:51)  at
kafka.Kafka$$anon$1.run(Kafka.scala:63)[2018-05-10 14:23:59,867] INFO KafkaConfig values:

from /var/log/kafka/kafka.err

Exception in thread "metrics-meter-tick-thread-3"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-2"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-5"
java.lang.OutOfMemoryError: Java heap spaceException: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"kafka-socket-acceptor-PLAINTEXT-6667"Exception in thread "metrics-meter-tick-thread-4"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-7"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-6"
java.lang.OutOfMemoryError: Java heap space

we not get any output from the following: ( port isn't licensing )

netstat -tnlpa | grep 6667

[#2]

The recommendation here would be to increase the heap space allocated to the Kafka process or reduce the amount of other processes running on the same server. For example, in a production environment, the Kafka brokers should be standalone servers -- not on the same hardware as Zookeeper or other Hadoop processes.

[#3]

@Jordan , regarding to what you said - "increase the heap space allocated to the Kafka process" can you give example of the parameter? , so we can find it from ambari GUI , second yes this kafka broker is stand alone machine and not with the Zookeeper

[#4]

I am not sure about the following , but do you mean to update the file - /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh and update the parameter - export KAFKA_HEAP_OPTS="-Xms2G -Xmx2G" ? ( according to the article - https://community.hortonworks.com/content/supportkb/151841/error-javalangoutofmemoryerror-direct-buf... )

[root@kafka01 conf]# more /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ $# -lt 1 ];
then
        echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
        exit 1
fi
base_dir=$(dirname $0)
if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi
EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}
COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    shift
    ;;
  *)
    ;;
esac
echo $KAFKA_HEAP_OPTS>>/tmp/uri
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

[#5]

@Jordan , another quastion please

regarding the file - /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh , I see that the default value - is KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" , so my question is , based on both variables that set to 1G , is it logical that 1G isn't enough ?

[#6]

@Jordan , another point , I forget to tell you that we also restart the kafka machine but this not help to resolve the kafka broker , so the option is to increase the value from 1G to 2G according to your solution ,

[#7]

@Michael Bronson

Kafka stores the latest offsets in memory before they are sent to disk, therefore, the more memory the better, with a max of 8G.

And I would assume that the heap properties can be set from Ambari rather than individually on the broker, but I don't use Kafka from HDP, so I can't say.