@Sriram    When a record is added to a batch, there is a time limit for sending that batch to ensure it has been sent within a specified duration controlled by request.timeout.ms.    Default is 30 seconds, you can try increasing this value and monitor. Can you share processor configurations to get more details?   ... View more

@sagarshimpi    I believe debug for worker nodes its fine for the storm side. From the Kafka side, we can review the broker .log files usually located under /var/log/kafka/server.log. Also, you can monitor if there are any under replicated partitions by running the command line below:     bin/kafka-topics.sh $ZK --describe --under-replicated-partitions   ... View more

@sagarshimpi    So if I understood correctly the issue is not occurring now, right? If not, then this could be related to a connection issue:   [ERROR] Failed to reconnect to cluster (consider increasing 'ne tworkTimeout' configuration property) [networkTimeout=5000]     ... View more

@sampathkumar_ma    export KAFA_OPTS should work in this case. Could you please add "debug=true" to the jaas file:   KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/<user>/user.keytab" storeKey=true debug=true useTicketCache=false serviceName="kafka" principal="user@domain.COM"; };   Share the complete output, we should see something similar to: Debug is true storeKey false useTicketCache true useKeyTab false doNotPrompt false ticketCache is null isInitiator true KeyTab is null refreshKrb5Config is false principal is null tryFirstPass is false useFirstPass is false storePass is false clearPass is false Acquire TGT from Cache Principal is kafka/host@EXAMPLE.COM Commit Succeeded Also along with that you can enable DEBUG under:  /etc/kafka/conf/tools-log4j.properties   Change WARN to DEBUG and run the client and share the details.       ... View more

@sampathkumar_ma    Could you please check if the user running the command has permissions to get a valid ticket from: "/home/<user>/user.keytab"   Also, add "debug=true" in the jaas file to get more details:   KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/home/<user>/user.keytab" storeKey=true debug=true useTicketCache=false serviceName="kafka" principal="user@domain.COM"; };   ... View more

@mike_bronson7    This could be related to a number of reasons, one of them is when moving Kafka data manually (full disk issues for example or replacing disks). In addition to that, there is a property called  unclean.leader.election.enable that in kafka HDP 2.6.4 is false by default:   unclean.leader.election.enable Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss   If you change this property to true and then to false without all brokers in sync, it also may cause the issue you have mentioned. If you want  durability over availability, then make sure this property is set to false. ... View more

@sagarshimpi    From the log files, I can see that the coordinator is dead:   2019-11-21 07:37:43.750 o.a.k.c.c.i.AbstractCoordinator Thread-22-ciFileSpout-executor[91 91] [INFO] [Consumer clientId=consumer-1, groupId=acp_c1_prv_input_client_instruction- consumer] Marking the coordinator testnode8o.example.com:6667 (id: 2147482643 rack: null) dead     Can you share the describe command output:   bin/kafka-topics.sh --describe --zookeeper <zkHost>:<zkPort>     Also, check if you're able to produce/consume messages in Kafka by using Kafka clients:     bin/kafka-console-producer.sh --broker-list testnode8o.example.com:6667 --topic <topicName>       bin/kafka-console-consumer.sh --bootstrap-server testnode8o.example.com:6667 --topic <topicName> --from-beginning   ... View more

@Adarsh_ks    Are you facing the same issue after the changes? kindly share the exception if different from the original.   Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=false keyTab="<keytab_path>" serviceName="zookeeper" principal="<principal>"; };   Make sure that the keytab and principal specified are accessible from the user that is running the job. ... View more

@sandeep_hadoopa    I would say that the logic has to be in the application. If this is because of fault-tolerant or high availability, you can use 1 single cluster with multiple hosts and replication factor >= 3 to replicate the topic's data among hosts. If one broker is down, then you have other 2 machines to replace the topic leader and continue working.     ... View more

HI @sandeep_hadoopa    I believe you can use Kafka mirror maker for this scenario, where you can write data to one cluster and replicate the messages to the other.    https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/kafka-mirroring-data/content/mirroring_data_between_clusters__using_the_mirrormaker_tool.html   I hope that helps. Regards, Manuel. ... View more

Hi @koshay    Can you provide more details about the exception posted? As far as I see, you're running a topology that is producing data to Kafka and you're seeing the exception mentioned, right? Kindly provide as much as details you can to get an idea on this issue.   Thanks. Manuel. ... View more

Hi @Adarsh_ks    Can you try adding to Client    serviceName="zookeeper"     Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true useTicketCache=false keyTab="<keytab_path>" serviceName="zookeeper" principal="<principal>"; }; Let us know the results. Thanks, Manuel.  ... View more

@anbazhagan_muth    You can leave them blank unless you're using mirror maker, in that case, you have to fill up those values using:   Destination Broker List bootstrap.servers     Comma-separated list of IP:port (or hostname:port) pairs of brokers on destination cluster. This should be more than one, for high availability, but there's no need to list all brokers.     Source Broker List source.bootstrap.servers     Comma-separated list of IP:port (or hostname:port) pairs of brokers on source cluster. This should be more than one, for high availability, but there's no need to list all brokers.       Documentation below: https://docs.cloudera.com/documentation/enterprise/5-7-x/topics/cm_props_cdh540_kafka.html#concept_5.7.x_kafkabroker_props     ... View more

@kiranps11    Can you try:    hostname -f   After that, from the output, use the machine name along with the ports.   Let us know how it goes. ... View more

@mike_bronson7 can you check if the meta.properties file usually located under /kafka-logs has the right value? this should contain broker.id=1   We can't add ids manually to zookeeper, these are ephemeral nodes and they're created every time that a broker registers itself in zookeeper.   Can you provide the startup lines from the server.log?   Thanks. ... View more

Hi @iamabug    Before running the consumer, could you please add to your terminal:   export KAFKA_OPTS="-Dsun.security.krb5.debug=true"   After that, share the DEBUG info for further review.   Thanks. ... View more

Hello @nhemamalini    Are you able to send/consume data to/from the topics by using the Kafka command line?   Producer: bin/kafka-console-producer.sh --broker-list node1:6667 --topic <topicName>   Consumer: bin/kafka-console-consumer.sh --bootstrap-server node1:6667 --topic <topicName> --from-beginning   Thanks.   ... View more

Hi @BaluSaiD,   If the topics you're producing/consuming data to/from have at least 2/3 in-sync replicas and min.insync.replicas=2, then it should be ok. If some topics have just 1 replica, and this broker dies, then you will not be able to produce/consume data from this topic.   Properties to keep in mind:   1. Server-side: min.insync.replicas: When a producer sets acks to "all" (or "-1"), min.insync.replicas specify the minimum number of replicas that must acknowledge a write for the write to be considered successful. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend). When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". This will ensure that the producer raises an exception if a majority of replicas do not receive a write.   2. Producer Side: ack: The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are allowed: acks=0 If set to zero then the producer will not wait for any acknowledgment from the server at all. The record will be immediately added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1. acks=1 This will mean the leader will write the record to its local log but will respond without awaiting full acknowledgment from all followers. In this case, should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost. acks=all This means the leader will wait for the full set of in-sync replicas to acknowledge the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. This is equivalent to the acks=-1 setting.   Regards, Manuel. ... View more

Thanks, @kiranps11 .   Is Nifi running in the same host than Kafka? If yes, can you specify IP addresses for the processor instead of localhost and see how it goes?   If not, make sure that the ports are listening properly by using netstat command for example.   In nifi-app you should be able to see the producer configurations, make sure that the broker list match with the broker's hosts and security protocol (PLAINTEXT).   ... View more

Hello,   Please check the Kafka ACLs and make sure that the ANONYMOUS user has permissions to write data to this topic.   You can try executing below command:   bin/kafka-acls.sh --authorizer-properties zookeeper.connect= localhost:2181 --add --allow-principal User:ANONYMOUS --operation All --topic=test --group=*   Regards, Manuel. ... View more