Reply
Explorer
Posts: 6
Registered: ‎06-02-2019

Kafka Broker service not starting

[ Edited ]

 

Hi, 

 

I am getting below error when starting Kafka Broker service. need help in this regard. 

 

kafka broker error.JPG

 

 

 

 

 

 

Sun Jun  2 03:39:15 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid3839.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:16 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,
Sun Jun  2 03:39:23 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid4586.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:23 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,
Sun Jun  2 03:39:36 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid5490.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:36 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,
Sun Jun  2 03:39:46 PDT 2019
JAVA_HOME=/usr/java/jdk1.8.0_211/
Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-6f7e9fa2e076f5332a81ffe11109ab36_pid6319.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS
Using /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER as conf dir
Using scripts/control.sh as process script
CONF_DIR=/var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CMF_CONF_DIR=/etc/cloudera-scm-agent

Date: Sun Jun  2 03:39:47 PDT 2019
Host: quickstart.cloudera
Pwd: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
CONF_DIR: /var/run/cloudera-scm-agent/process/82-kafka-KAFKA_BROKER
KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka
Zookeeper Quorum: quickstart.cloudera:2181
Zookeeper Chroot: quickstart.cloudera
PORT: 9092
JMX_PORT: 9393
SSL_PORT: 9093
ENABLE_MONITORING: true
METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter
BROKER_HEAP_SIZE: 256
BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true
BROKER_SSL_ENABLED: false
KERBEROS_AUTH_ENABLED: false
KAFKA_PRINCIPAL: 
SECURITY_INTER_BROKER_PROTOCOL: INFERRED
AUTHENTICATE_ZOOKEEPER_CONNECTION: true
SUPER_USERS: kafka
Kafka version found: 0.10.0-kafka2.1.0
Sentry version found: 1.5.1-cdh5.13.0
ZK_PRINCIPAL_NAME: zookeeper
Final Zookeeper Quorum is quickstart.cloudera:2181quickstart.cloudera
security.inter.broker.protocol inferred as PLAINTEXT
LISTENERS=listeners=PLAINTEXT://quickstart.cloudera:9092,

 

New Contributor
Posts: 2
Registered: ‎06-03-2019

Re: Kafka Broker service not starting

Can you Add Role Log ?

Explorer
Posts: 6
Registered: ‎06-02-2019

Re: Kafka Broker service not starting

sure, please find below. 

 

metrics.num.samples = 2
	socket.send.buffer.bytes = 102400
	ssl.protocol = TLS
	socket.receive.buffer.bytes = 102400
	ssl.keystore.location = null
	replica.fetch.min.bytes = 1
	broker.rack = null
	unclean.leader.election.enable = false
	sasl.enabled.mechanisms = [GSSAPI]
	group.min.session.timeout.ms = 6000
	log.cleaner.io.buffer.load.factor = 0.9
	offsets.retention.check.interval.ms = 600000
	producer.purgatory.purge.interval.requests = 1000
	metrics.sample.window.ms = 30000
	broker.id = 46
	offsets.topic.compression.codec = 0
	log.retention.check.interval.ms = 300000
	advertised.listeners = null
	leader.imbalance.per.broker.percentage = 10

2019-06-08 10:26:43,280 INFO kafka.server.KafkaServer: starting
2019-06-08 10:26:43,290 INFO kafka.server.KafkaServer: Connecting to zookeeper on quickstart.cloudera:2181
2019-06-08 10:26:43,296 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=quickstart.cloudera:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@4b40f651
2019-06-08 10:26:43,297 INFO org.I0Itec.zkclient.ZkEventThread: Starting ZkClient event thread.
2019-06-08 10:26:43,300 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server quickstart.cloudera/172.16.0.40:2181. Will not attempt to authenticate using SASL (unknown error)
2019-06-08 10:26:43,302 INFO org.apache.zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.16.0.40:41453, server: quickstart.cloudera/172.16.0.40:2181
2019-06-08 10:26:43,303 INFO org.I0Itec.zkclient.ZkClient: Waiting for keeper state SyncConnected
2019-06-08 10:26:43,303 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server quickstart.cloudera/172.16.0.40:2181, sessionid = 0x16b17eb50c88c36, negotiated timeout = 6000
2019-06-08 10:26:43,304 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected)
2019-06-08 10:26:43,647 INFO kafka.log.LogManager: Loading logs.
2019-06-08 10:26:43,667 INFO kafka.log.LogManager: Logs loading complete.
Highlighted
Explorer
Posts: 6
Registered: ‎06-02-2019

Re: Kafka Broker service not starting

I have manged to start Kafka Broker, once I changed "Java Heap Size of Broker" from 50MB to 1024MB.

 

Thanks and Best Regards, 

 

Announcements