Member since
08-08-2017
1652
Posts
30
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1953 | 06-15-2020 05:23 AM | |
| 15901 | 01-30-2020 08:04 PM | |
| 2097 | 07-07-2019 09:06 PM | |
| 8195 | 01-27-2018 10:17 PM | |
| 4652 | 12-31-2017 10:12 PM |
05-13-2018
03:20 PM
@Geoffrey thank you I will verify this in the next days , and I will update , this is very strange because this happens only on one kafka , while other two kafka's machine are ok
... View more
05-13-2018
03:11 PM
@Geoffrey just note - the machine isnt rebooted ! , kafka broker is restating - its means that from ambari GUI you can see that kafka broker is up / down and finnaly down , but kafka linux machine stay up without reboot
... View more
05-13-2018
11:12 AM
hi all we have hadoop cluster version - 2.6.4 , and separate kafka machines ( 3 kafka machines ) on of the kafka machines ( kafka03 ) is restarting all the time and in some time could be stooped from netstat we can see that kafka cant listening to port 6667 we check the /var/log/kafka.err file , but this file is empty after we clean it by cp /dev/null /var/log/kafka.err and start the kafka broker to see if any info will comes to /var/log/kafka.err any other ideas how to understand why kafka machine not listening to port 6667? 76.12.76.73 is the IP of kafka03 server
76.12.76.74 is the IP of kafka01 server netstat -tnlpa | grep 6667
tcp6 0 0 76.12.76.73:43612 76.12.76.74:6667 ESTABLISHED 94962/java
tcp6 0 0 76.12.76.73:43616 76.12.76.74:6667 ESTABLISHED 94962/java
tcp6 0 0 76.12.76.73:43613 76.12.76.74:6667 ESTABLISHED 94962/java
netstat -tnlpa | grep 6667
tcp6 0 0 76.12.76.73:43616 76.12.76.74:6667 ESTABLISHED 94962/java
tcp6 0 0 76.12.76.73:43613 76.12.76.74:6667 ESTABLISHED 94962/java
... View more
Labels:
05-11-2018
01:56 PM
@Jordan , another point , I forget to tell you that we also restart the kafka machine but this not help to resolve the kafka broker , so the option is to increase the value from 1G to 2G according to your solution ,
... View more
05-11-2018
12:29 PM
@Jordan , another quastion please regarding the file - /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh , I see that the default value - is KAFKA_HEAP_OPTS="-Xmx1G -Xms1G" , so my question is , based on both variables that set to 1G , is it logical that 1G isn't enough ?
... View more
05-11-2018
05:50 AM
I am not sure about the following , but do you mean to update the file - /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh and update the parameter - export KAFKA_HEAP_OPTS="-Xms2G -Xmx2G" ? ( according to the article - https://community.hortonworks.com/content/supportkb/151841/error-javalangoutofmemoryerror-direct-buffer-memor.html ) [root@kafka01 conf]# more /usr/hdp/2.6.4.0-91/kafka/bin/kafka-server-start.sh
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ $# -lt 1 ];
then
echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
exit 1
fi
base_dir=$(dirname $0)
if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi
EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}
COMMAND=$1
case $COMMAND in
-daemon)
EXTRA_ARGS="-daemon "$EXTRA_ARGS
shift
;;
*)
;;
esac
echo $KAFKA_HEAP_OPTS>>/tmp/uri
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
... View more
05-11-2018
04:42 AM
@Jordan , regarding to what you said - "increase the heap space allocated to the Kafka process" can you give example of the parameter? , so we can find it from ambari GUI , second yes this kafka broker is stand alone machine and not with the Zookeeper
... View more
05-10-2018
04:57 PM
we have in our Hadoop cluster 3 kafka brokers ( based on ambari ) one of the kafka broker can’t starting ( kafka01 ) any suggestion for this situation ? we have the following logs from
/var/log/kafka/server.log from /var/log/kafka/server.log: FATAL Fatal error<br>
during KafkaServer shutdown. (kafka.server.KafkaServer)java.lang.IllegalStateException: Kafka server is still
starting up, cannot shut down! at
kafka.server.KafkaServer.shutdown(KafkaServer.scala:576) at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:51) at
kafka.Kafka$$anon$1.run(Kafka.scala:63)[2018-05-10 14:23:57,032] FATAL Fatal error during
KafkaServerStable shutdown. Prepare to halt (kafka.server.KafkaServerStartable)java.lang.IllegalStateException: Kafka server is still
starting up, cannot shut down! at
kafka.server.KafkaServer.shutdown(KafkaServer.scala:576) at
kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:51) at
kafka.Kafka$$anon$1.run(Kafka.scala:63)[2018-05-10 14:23:59,867] INFO KafkaConfig values: from /var/log/kafka/kafka.err Exception in thread "metrics-meter-tick-thread-3"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-2"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-5"
java.lang.OutOfMemoryError: Java heap spaceException: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread
"kafka-socket-acceptor-PLAINTEXT-6667"Exception in thread "metrics-meter-tick-thread-4"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-7"
java.lang.OutOfMemoryError: Java heap spaceException in thread "metrics-meter-tick-thread-6"
java.lang.OutOfMemoryError: Java heap space we not get any output from the following: ( port isn't licensing ) netstat -tnlpa | grep 6667
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
-
Apache Kafka
05-01-2018
02:35 PM
just for summary ( this is production system ) the final steps are
hdfs fsck / -delete and if "step 1" not fixed the corrupted blocks then we need to remove the file as: hdfs fs -rm /localF/STRZONEZone/intercept_by_country/2018/4/10/16/2018_4_10_16_45.parquet/part-00003-8600d0e2-c6b6-49b7-89cd-ef2a2bc1dc5e.snappy.parquet is it correct ?
... View more
04-30-2018
04:00 PM
we have ambari cluster with HDP version 26 ( production system ) when we run the following command in order to verify which files have corrupted blocks hdfs fsck / |egrep -v '^\.+$' | grep -v replica | grep -v Replica we get: /localF/STRZONEZone/intercept_by_country/2018/4/10/16/2018_4_10_16_45.parquet/part-00003-8600d0e2-c6b6-49b7-89cd-ef2a2bc1dc5e.snappy.parquet: CORRUPT blockpool BP-338831142-28.12.45.6-1508451686931 block blk_1097240348
/localF/STRZONEZone/intercept_by_country/2018/4/10/16/2018_4_10_16_45.parquet/part-00003-8600d0e2-c6b6-49b7-89cd-ef2a2bc1dc5e.snappy.parquet: MISSING 1 blocks of total size 1192 B...........................................
/localF/STRZONEZone/intercept_by_type/2018/4/10/16/2018_4_10_16_45.parquet/part-00002-be0f80a9-2c7c-4c50-b18d-73be372acff.snappy.parquet: CORRUPT blockpool BP-338831142-28.12.45.6-1508451686931 block blk_1097240344
/localF/STRZONEZone/intercept_by_type/2018/4/10/16/2018_4_10_16_45.parquet/part-00002-be0f80a9-2c7c-4c50-b18d-73be372acff.snappy.parquet: MISSING 1 blocks of total size 1098 B...............................................
..................................Status: CORRUPT
Total size:7072689634566 B (Total open files size: 293676105509 B)
Total dirs:32330710
Total files:910568034
Total symlinks:0 (Files currently being written: 12)
Total blocks (validated):10183608 (avg. block size 6254517 B) (Total open file blocks (not validated): 2200)
********************************
UNDER MIN REPL'D BLOCKS:2 (1.9345605E-5 %)
CORRUPT FILES:2
MISSING BLOCKS:2
MISSING SIZE:2290 B
CORRUPT BLOCKS: 2
********************************
Corrupt blocks:2
Number of data-nodes:35
Number of racks:1
FSCK ended at Mon Apr 20 11:40:50 UTC 2018 in 241684 milliseconds
The filesystem under path '/' is CORRUPT
in this case that we see : CORRUPT FILES:2
MISSING BLOCKS:2 what is the right action to do ? , or corrupted blocs solutuion ?
... View more
Labels: