Thanks for your reply. In our case we're operating the kafka cluster not on CDP. Consuming data from kafka is no problem. Please have a look at this "org.apache.zookeeper.Login: TGT renewal thread ha... - Cloudera Community - 373868 ... View more

Hello @AndreyKravtsov    Please refer to the article below to Integrating Apache NiFi and Apache Kafka [1]   [1]  https://community.cloudera.com/t5/Community-Articles/Integrating-Apache-NiFi-and-Apache-Kafka/ta-p/247433   This example is with the PLAINTEXT Kafka protocol. It looks like you are using SSL/TLLS for kafka. You should check whether you are using SSL or SASL_SSL protocol for kafka from CM >> Kafka conf and created a StandardSSLContextService controller setting and update with Kafka Keystore and Truststore details. You can refer the below article for details [2]:   [2] https://community.cloudera.com/t5/Support-Questions/Need-help-with-SSL-config-in-Nifi-ConsumeKafka/td-p/320594   Additionally, refer the following articles for more details:  [1] - https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.17.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html [2] - https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.20.0/org.apache.nifi.ssl.StandardSSLContextService/index.html   If this information helped you, then it will be appreciated if you will take a moment to click on KUDOS 🙂 ... View more

@yamu Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks! ... View more

@Babasaheb  Appreciate all you assistance  ... View more

 @DataMike    Yes, you can use the CC APIs as per your requirements [1] [2]   [1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cctrl-managing/topics/cctrl-using-rest-api.html [2] https://community.cloudera.com/t5/Customer/Frequently-Used-CRUISE-CONTROL-API-and-important-DOCs/ta-p/324729   From the above articles you can use the following API to rebalance topics/partitions:    curl -k --negotiate -u: -X POST "https://<CC FQDN>:8899/kafkacruisecontrol/rebalance?dryrun=false&rebalance_disk=true"   To avoid high CPU, memory and disk read write you can initiate rebalance process in batches. It will automatically create a batch and rebalance topics.   curl -X POST  "http://$HOSTNAME:8899/kafkacruisecontrol/rebalance?dryrun=true&concurrent_partition_movements_per_broker=10&concurrent_leader_movements=500"   If it will help you then please click on ”Accept as Solution" below this post.   Thank you. ... View more

@mohammad_shamim Did you have Hive HA configured in CDH cluster, in that case, you need to make sure that there are equal number of HS2 instances created in the CDP cluster, because without that HA cannot be attained. Also, make sure that there is no Hiveserver2 instance created under "Hive" service in CDP. It should only be present under Hive on Tez service. ... View more

@Onkar_Gagre    I would suggest you to tweak the following settings for better MirrorMaker scaling:   1) Kafka > Config > consumer.request.timeout.ms to 500000 Kafka > Config > session.timeout.ms to 60000 Kafka > Config > buffer.memory to 66000000 Kafka > Config > replica.fetch.max.bytes to 56700160 Kafka > Config > message.max.bytes to 56700160   2) You can increase the heartbeat.interval.ms to 20 secs in line with session.timeout.ms (1/3rd of session time). This has to be set in CM > Kafka > Config > Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties > heartbeat.interval.ms = 20000 > Save changes > Restart MM.   3) Add   partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor   to Kafka safety value for mirror.maker.consumer.properties   If you still face the same issue, then provide us the following details for further investigation:   1) Are you getting any errors in the broker logs and MM logs? OR we need broker and MM logs  2) Run in each broker on the destination cluster a couple of times every 5 hours below to discard networking issues: netstat -s | egrep "retransmited|sent out|socket|buffer|overflow|total packets received" 3) How many brokers are there in the Kafka cluster?  4) Was it working fine before and have you made any recent changes?    Thank you. ... View more

@Onkar_Gagre    A] There is no any standard tool that Kafka or Cloudera provides to migrate only Kafka zookeeper metadata. But yes, there are some third-party tools which you can use to migrate zookeeper metadata.    You can use a third-party tool like zkcopy to migrate/copy metadata from one cluster to another.  But we would recommend testing it in your dev lab first to avoid further issues    Note: Just something to be aware of.. If you decide on copying over the znodes uses this third party tool then you need to do this when the Kafka service is shutdown. You would then copy it to the new Znode root location, change the chroot config in Kafka and start that up again.   B] You can use SRM for migration and once the migration is completed you can lower the retention settings 1-2 hours as you don't need data so that all data will be cleared automatically and then you can set retetion back to 7 days which is default or as per your requirement.    C] You can also create all topics on a new cluster with the required configuration    If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.   Thank you. ... View more