Cloudera Community
Announcements
Now Live: Explore expert insights and technical deep dives on the new Cloudera Community Blogs — Read the Announcement
Shelton
user rank
Master Mentor

Re: validate Kafka reassignment for partitions

by Master Mentor in Support Questions
‎01-24-2019 06:18 PM
1 Kudo
‎01-24-2019 06:18 PM
1 Kudo
@Michael Bronson How to Rebalance Topics in a Kafka Cluster Here is a superb reference by Upendra Mutori to sort out your current issue rebalancing kafka topics Kafka Monitoring Tools Any monitoring tools with JMX support should be able to monitor a Kafka cluster. Here are some monitoring tools : First one is check_kafka.pl from Hari Sekhon. It performs a complete end to end test, i.e. it inserts a message in Kafka as a producer and then extracts it as a consumer. This makes our life easier when measuring service times. Another useful tool is KafkaOffsetMonitor for monitoring Kafka consumers and their position (offset) in the queue. It aids our understanding of how our queue grows and which consumers groups are lagging behind. Last but not least, the LinkedIn folks have developed what I think is the smartest tool out there: Burrow. It analyzes consumer offsets and lags over a window of time and determines the consumer status. You can retrieve this status over an HTTP endpoint and then plug it into your favourite monitoring tool (Server Density for example). There is also Yahoo’s Kafka-Manager. While it does include some basic monitoring, it is more of a management tool. If you are just looking for a Kafka management tool, check out AirBnb’s kafkat. HTH ... View more

Re: namenode down?

by Master Mentor in Support Questions
‎01-24-2019 04:07 PM
‎01-24-2019 04:07 PM
@Lokesh Mukku Good to know it has given you a better understanding. If you found this answer addressed your question, please take a moment to log in and click the "accept" link on the answer. That would be a great help to Community users to find the solution quickly for these kinds of errors. Happy hadooping !!!!! ... View more

Re: kafka + Exception: about Address already in us...

by Master Mentor in Support Questions
‎01-24-2019 02:38 PM
1 Kudo
‎01-24-2019 02:38 PM
1 Kudo
@Michael Bronson Zookeeper's usually have the same port so I suggest you have a look at the brokers. Please have a look at my response to a similar problem https://community.hortonworks.com/questions/235766/kafka-socket-server-failed-bind-exception.html?childToView=234784#answer-234784 HTH ... View more

Re: namenode down?

by Master Mentor in Support Questions
‎01-24-2019 08:16 AM
‎01-24-2019 08:16 AM
@Lokesh Mukku In a Namenode HA there is no notion of IP address but Nameservice ID (geeklab)as illustrated in the screenshot. In the background, geeklab will be doing something like load balancing between (nn01,nn02) whichever is the Active at a particular time, for example, if nn01 is the active NN and all over a sudden it encounters an issue and goes down, the ZKFailoverController will detect that failure through the heartbeat misses and transparently failover to nn02. The Nameservice ID is used to avoid hard-coding the IP's in case you change the host during an upgrade or system failure. I gave you the example of DNS to explain it further..www.teamworks.com is a DNS entry used to resolve hostnames to a specific IP of the teamworks.com web server to machine-readable IP addresses (e.g. 192.168.106.81) behind the scenes www.teamworks.com to guard against any failure could have a couple of web servers (HA setup ) say 3 web servers serving content for www.teamworks.com with a load balancer (see screenshot LB.png) which act like the Nameservice ID in case a namenode/web server goes down the cluster/website will still be available as Nameservice ID will automatically failover point to the active name node controlled by ZKFailoverController . HTH ... View more

Re: namenode down?

by Master Mentor in Support Questions
‎01-23-2019 06:02 PM
1 Kudo
‎01-23-2019 06:02 PM
1 Kudo
@Lokesh Mukku The sequence of events you described is correct. The failover should be transparently handled by the ZKFailoverController . HA clusters use the nameservice ID (dfs.nameservices in hdfs-site.xml) to identify a single HDFS instance that may consist of multiple HA NameNodes. The nameservice ID acts like a DNS for further reading see this HW document HTH ... View more

Re: Kafka Socket server failed - Bind Exception

by Master Mentor in Support Questions
‎01-23-2019 05:39 PM
1 Kudo
‎01-23-2019 05:39 PM
1 Kudo
@Mahendiran Palani Samy That's typical a port conflict issue "Socket server failed to bind to 0.0.0.0:9092:Address already inuse" Kafka listens on TCP port 9092 so this means there is already a process running on port 9092 change the port KafkaServer id=0 to something like 9093 or get the PID running on the port and kill it then restart kafka # lsof -n -i :9092 | grep LISTEN The PID should be the second column entry eg 7812 below java 7812 root 205u IPv6 60200 0t0 TCP *:webcache (LISTEN) Now kill the process # kill -9 7812 Now your Kafka should restart successfully , if you are running multiple brokers do configure like below broker.id=0 listeners=PLAINTEXT://:9092 log.dirs=/tmp/kafka-logs server.1.properties broker.id=1 listeners=PLAINTEXT://:9093 log.dirs=/tmp/kafka-logs1 server.2.properties broker.id=2 listeners=PLAINTEXT://:9094 log.dirs=/tmp/kafka-logs2 server.3.properties broker.id=3 listeners=PLAINTEXT://:9095 log.dirs=/tmp/kafka-logs3 ... View more

Re: ConsumeKafka processor's NiFi can't communicat...

by Master Mentor in Support Questions
‎01-23-2019 04:55 PM
‎01-23-2019 04:55 PM
@Ruslan Fialkovsky I think the topics created before changing the ACL's don't inherit the permissions. From the znode entry, I see you have one topic called test. To validate can you create a new topic and ensure you change them in your Nifi flow files and retest! Normally if you run rmr in zookeeper CLI the entry deleted should be created but I am not sure for Kafka topics. Please revert ... View more

Re: ConsumeKafka processor's NiFi can't communicat...

by Master Mentor in Support Questions
‎01-23-2019 10:07 AM
‎01-23-2019 10:07 AM
@ Ruslan Fialkovsky Logon to Zk on DEV and PROD Check whether you have child znodes under ls /config/topics Validate that you have Kerberos entries in /etc/nifi/conf/nifi_jaas.conf example NiFiClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/nifi.service.keytab" storeKey=true useTicketCache=false principal="nifi/{kdc_host}@REALM"; }; RegistryClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/nifi.service.keytab" storeKey=true useTicketCache=false principal="nifi/{kdc_host}@REALM"; Please revert ... View more

Re: ConsumeKafka processor's NiFi can't communicat...

by Master Mentor in Support Questions
‎01-23-2019 08:19 AM
‎01-23-2019 08:19 AM
@ Ruslan Fialkovsky Can you compare these files on both cluster, if it doesn't exist on the DEV then create them with correct owner and privileges Kafka /etc/kafka/conf/kafka_client_jaas.conf /etc/kafka/conf/kafka_jaas.conf Zookeeper /etc/zookeeper/conf/zookeeper_jaas.conf /etc/zookeeper/conf/zookeeper_client_jaas.conf After updating these files restart the zk and Kafka and retry HTH ... View more

Re: ConsumeKafka processor's NiFi can't communicat...

by Master Mentor in Support Questions
‎01-23-2019 07:06 AM
‎01-23-2019 07:06 AM
@ Ruslan Fialkovsky Now you see that on DEV world: everyone has cdrwa while for kafka there is no entry so you will need to setAcl for kafka setAcl /config/topics world:anyone:cdrwa,sasl:kafka:cdrwa Did you grasp how to use the using superDigest to become a Zookeeper superuser, that's like the zookeeper root user who can add/change the ACL's ,let me know if you need help on that. ... View more
Contact Me
Online
Offline
Last Visited
‎12-11-2025 11:50 PM
Community Statistics
Member Since ‎01-19-2017 04:35 AM
Last Visited ‎12-11-2025 11:50 PM
Posts 3,679
Kudos received 627