Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka Broker service not starting

Kafka Broker service not starting

Explorer

 

Hi, 

 

I am getting below error when starting Kafka Broker service. need help in this regard. 

 

kafka broker error.JPG

 

 

 

 

 

 

Sun Jun  2 03:39:15 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid3839.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:16 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,
Sun Jun  2 03:39:23 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid4586.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:23 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,
Sun Jun  2 03:39:36 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid5490.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:36 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,
Sun Jun  2 03:39:46 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid6319.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:47 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,

 

3 REPLIES 3

Re: Kafka Broker service not starting

New Contributor

Can you Add Role Log ?

Re: Kafka Broker service not starting

Explorer

sure, please find below. 

 

metrics.num.samples = 2
	socket.send.buffer.bytes = 102400
	ssl.protocol = TLS
	socket.receive.buffer.bytes = 102400
	ssl.keystore.location = null
	replica.fetch.min.bytes = 1
	broker.rack = null
	unclean.leader.election.enable = false
	sasl.enabled.mechanisms = [GSSAPI]
	group.min.session.timeout.ms = 6000
	log.cleaner.io.buffer.load.factor = 0.9
	offsets.retention.check.interval.ms = 600000
	producer.purgatory.purge.interval.requests = 1000
	metrics.sample.window.ms = 30000
	broker.id = 46
	offsets.topic.compression.codec = 0
	log.retention.check.interval.ms = 300000
	advertised.listeners = null
	leader.imbalance.per.broker.percentage = 10

2019-06-08 10:26:43,280 INFO kafka.server.KafkaServer: starting
2019-06-08 10:26:43,290 INFO kafka.server.KafkaServer: Connecting to zookeeper on quickstart.cloudera:2181
2019-06-08 10:26:43,296 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@4b40f651
2019-06-08 10:26:43,297 INFO org.I0Itec.zkclient.ZkEventThread: Starting ZkClient event thread.
2019-06-08 10:26:43,300 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/172.16.0.40:2181. Will not attempt to authenticate using SASL (unknown error)
2019-06-08 10:26:43,302 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.16.0.40:41453, server: quickstart.cloudera/172.16.0.40:2181
2019-06-08 10:26:43,303 INFO org.I0Itec.zkclient.ZkClient: Waiting for keeper state SyncConnected
2019-06-08 10:26:43,303 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/172.16.0.40:2181, sessionid = 0x16b17eb50c88c36, negotiated timeout = 6000
2019-06-08 10:26:43,304 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected)
2019-06-08 10:26:43,647 INFO kafka.log.LogManager: Loading logs.
2019-06-08 10:26:43,667 INFO kafka.log.LogManager: Logs loading complete.
Highlighted

Re: Kafka Broker service not starting

Explorer

I have manged to start Kafka Broker, once I changed "Java Heap Size of Broker" from 50MB to 1024MB.

 

Thanks and Best Regards,