Hi @Peruvian81    It's difficult to suggest if there are no details about the cluster usage, but it would be useful to start reviewing the below article that provides Kafka best practices.   https://community.cloudera.com/t5/Community-Articles/Kafka-Best-Practices/ta-p/249371   I hope that helps. Regards, Manuel. ... View more

Hi @Seaport    If you're using ambari,  Enable Atlas Hook should take case of that. In addition to that,  follow the steps below:   cp /usr/hdp/current/atlas-server/conf/atlas-application.properties /etc/hbase/conf get a valid ticket from atlas user export HBASE_CONF_DIR=/usr/hdp/current/hbase-client/conf In ambari > hbase > advanced hbase-site add: hbase.coprocessor.master.classes=org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor, org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor (Restart required). Finally run: /usr/hdp/current/atlas-server/hook-bin/import-hbase.sh             ... View more

@anbazhagan_muth    You can leave them blank unless you're using mirror maker, in that case, you have to fill up those values using:   Destination Broker List bootstrap.servers     Comma-separated list of IP:port (or hostname:port) pairs of brokers on destination cluster. This should be more than one, for high availability, but there's no need to list all brokers.     Source Broker List source.bootstrap.servers     Comma-separated list of IP:port (or hostname:port) pairs of brokers on source cluster. This should be more than one, for high availability, but there's no need to list all brokers.       Documentation below: https://docs.cloudera.com/documentation/enterprise/5-7-x/topics/cm_props_cdh540_kafka.html#concept_5.7.x_kafkabroker_props     ... View more

@kiranps11    Can you try:    hostname -f   After that, from the output, use the machine name along with the ports.   Let us know how it goes. ... View more

@mike_bronson7 can you check if the meta.properties file usually located under /kafka-logs has the right value? this should contain broker.id=1   We can't add ids manually to zookeeper, these are ephemeral nodes and they're created every time that a broker registers itself in zookeeper.   Can you provide the startup lines from the server.log?   Thanks. ... View more

Hi @iamabug    Before running the consumer, could you please add to your terminal:   export KAFKA_OPTS="-Dsun.security.krb5.debug=true"   After that, share the DEBUG info for further review.   Thanks. ... View more

Hello @nhemamalini    Are you able to send/consume data to/from the topics by using the Kafka command line?   Producer: bin/kafka-console-producer.sh --broker-list node1:6667 --topic <topicName>   Consumer: bin/kafka-console-consumer.sh --bootstrap-server node1:6667 --topic <topicName> --from-beginning   Thanks.   ... View more

Hi @BaluSaiD,   If the topics you're producing/consuming data to/from have at least 2/3 in-sync replicas and min.insync.replicas=2, then it should be ok. If some topics have just 1 replica, and this broker dies, then you will not be able to produce/consume data from this topic.   Properties to keep in mind:   1. Server-side: min.insync.replicas: When a producer sets acks to "all" (or "-1"), min.insync.replicas specify the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend). When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.   2. Producer Side: ack: The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed: acks=0 If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1. acks=1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgment from all followers. In this case, should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost. acks=all This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.   Regards, Manuel. ... View more

Thanks, @kiranps11 .   Is Nifi running in the same host than Kafka? If yes, can you specify IP addresses for the processor instead of localhost and see how it goes?   If not, make sure that the ports are listening properly by using netstat command for example.   In nifi-app you should be able to see the producer configurations, make sure that the broker list match with the broker's hosts and security protocol (PLAINTEXT).   ... View more

Hello,   Please check the Kafka ACLs and make sure that the ANONYMOUS user has permissions to write data to this topic.   You can try executing below command:   bin/kafka-acls.sh --authorizer-properties zookeeper.connect= localhost:2181 --add --allow-principal User:ANONYMOUS --operation All --topic=test --group=*   Regards, Manuel. ... View more

Hello,    Regarding to your questions:   1. Does the FULL CLUSTER RESTART distribute the topic partitions evenly across 5 nodes? All the topics already created will remain the same. Or need to run the partition reassignment tool kafka-reassign-partitions.sh ? Yes, this is the available approach. 2. Doesn't Kafka have an internal mechanism to automatically reassign the partitions unless running kafka-reassign-partitions.sh Unfortunately, this task has to be done manually by using the reassign partition tool.   You can check how this tool works by looking at the video article below: https://www.youtube.com/watch?v=TjKtCKsUjbQ   Hope this helps. Regards, Manuel.   ... View more

Hi Mike,    The controller is responsible for administrative opera tions, including assigning partitions to brokers and monitoring for broker failures.   If we are getting no leaders assigned, one thing you can check is the controller status in the zookeeper cli.   To perform ZooKeeper CLI operations you can use the ZooKeeper client by " bin/zkCli.sh" after that execute: get /controller and see if the output is showing an active controller. If the output is showing "null" you can run rmr /controller, this will trigger a new controller assignment. Finally, make sure you don't have authorization issues by checking the server.log files during broker restart.   Regards, Manuel.   ... View more

Hello,   Producers create new messages. In other publish/subscribe systems, these may be called publishers or writers. In general, a message will be produced to a specific topic. By default, the producer does not care what partition a specific message is written to and will balance messages over all partitions of a topic evenly. In some cases, the producer will direct messages to specific partitions. This is typically done using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. This assures that all messages produced with a given key will get written to the same partition.   If you want to distribute the data among partitions make sure no key or partition is specified in your client.   Hope this helps. Regards, Manuel. ... View more

Hi Saurav,  Can you please enable client DEBUG and see if the security.protocol is passed properly? also, it may help you also to get more detailed information about the timeout issue.   This can be done by editing below file:   /etc/kafka/conf/tools-log4j.properties   Change the following line from:   log4j.rootLogger=WARN, stderr To log4j.rootLogger=DEBUG, stderr   After that, run the consumer again and l et us know how it goes. Thanks, Manuel. ... View more

You can check the information in the article created by @Sriharsha Chintalapani below: https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html Hope this helps. ... View more

Could you please try to "describe" the affected topic using command below: kafka-topics.sh --describe --zookeeper <zkHost:zkPort> --topic TEST_KAFKA_TOPIC Please share the output. Are you able to consume from this topic? you can test by using kafka APIs below: kafka-console-consumer.sh --bootstrap-server <brokerHost:brokerPort> --topic TEST_KAFKA_TOPIC --new-consumer --from-beginning Note: If you're using kerberos, make sure to have a valid ticket and include --security-protocol SASL_PLAINTEXT Thanks. ... View more

In that case you have to upgrade your HDP version, latest HDP 2.6.5 release comes with Apache Kafka 1.0.0. Hope this helps. ... View more

Which is the command used to produce messages to this topic? Could you please use "describe" the topic that you're trying to produce messages?. You can use: ./kafka-topics.sh --describe --zookeeper <zkHost:zkPort> --topic <topicName> Thanks. ... View more

In order to list consumers, you can use: /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --zookeeper <zkHost>:<zkPort> --list Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API). /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --bootstrap-server <bokerHost:BrokerPort> --list Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). If you're using kerberos: /usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --bootstrap-server <bokerHost:BrokerPort> --list --new-consumer --command-config /tmp/test.properties cat/tmp/test.properties security.protocol=PLAINTEXTSASL If this answers your query/issue then please mark this HCC thread as answered by clicking on "Accept" link on the correct answer, That way it will help other HCC users to quickly find the answers. ... View more

Please edit "vi /etc/ambari-server/conf/ambari.properties" java.home=/usr/<jdk>/jdk_version After that restart ambari server. ... View more

1. Did you check the "describe" command for each topic? it matches with the brokers ids registered in zookeeper and meta.properties file? 2. Add --new-consumer to the command and try again ... View more