@yagoaparecidoti    You can add these properties as additional properties in “HDFS Client Configuration Safety Valve for hdfs-site.xml”. dfs.client.block.write.replace-datanode-on-failure.policy=ALWAYS dfs.client.block.write.replace-datanode-on-failure.best-effort=true   If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.   Thank you. ... View more

@cfoo    I see the following in the Nimbus log:   [ERROR] Error when processing event org.apache.storm.generated.KeyNotFoundException: null   Could you please try the below action plan and let us know how it goes:    - Increase the JVM of the nimbus from the existing value to keep the nimbus up for a few additional moments to kill the topologies. - stop all the topologies - delete the znode /storm ( From zk cli : rmr /storm ) - take a backup of storm.local.dir and delete the same( From zk cli : rm -rf <value of stormlocal.dir> ). You can check it in the storm.yaml file - start the supervisors and nimbus, check if the services are up and running - If there is no Nimbus service running, then the only way to remove the topology processes would be to manually kill the worker processes on all worker nodes.   If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.   Thank you. ... View more

@Onkar_Gagre    I would suggest you to tweak the following settings for better MirrorMaker scaling:   1) Kafka > Config > consumer.request.timeout.ms to 500000 Kafka > Config > session.timeout.ms to 60000 Kafka > Config > buffer.memory to 66000000 Kafka > Config > replica.fetch.max.bytes to 56700160 Kafka > Config > message.max.bytes to 56700160   2) You can increase the heartbeat.interval.ms to 20 secs in line with session.timeout.ms (1/3rd of session time). This has to be set in CM > Kafka > Config > Kafka MirrorMaker Advanced Configuration Snippet (Safety Valve) for mirror_maker_consumers.properties > heartbeat.interval.ms = 20000 > Save changes > Restart MM.   3) Add   partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor   to Kafka safety value for mirror.maker.consumer.properties   If you still face the same issue, then provide us the following details for further investigation:   1) Are you getting any errors in the broker logs and MM logs? OR we need broker and MM logs  2) Run in each broker on the destination cluster a couple of times every 5 hours below to discard networking issues: netstat -s | egrep "retransmited|sent out|socket|buffer|overflow|total packets received" 3) How many brokers are there in the Kafka cluster?  4) Was it working fine before and have you made any recent changes?    Thank you. ... View more

@radha56    We need some more details to check and investigate this issue further:    1) Provide us the stdout and stderr you are getting while restarting broker from Ambari >> Kafka  2)  Complete ERROR trace from the log file from failed broker node  3) Alert/WARN you are getting in Ambari >> Kafka >> Status page    Thank you.  ... View more

@Onkar_Gagre    A] There is no any standard tool that Kafka or Cloudera provides to migrate only Kafka zookeeper metadata. But yes, there are some third-party tools which you can use to migrate zookeeper metadata.    You can use a third-party tool like zkcopy to migrate/copy metadata from one cluster to another.  But we would recommend testing it in your dev lab first to avoid further issues    Note: Just something to be aware of.. If you decide on copying over the znodes uses this third party tool then you need to do this when the Kafka service is shutdown. You would then copy it to the new Znode root location, change the chroot config in Kafka and start that up again.   B] You can use SRM for migration and once the migration is completed you can lower the retention settings 1-2 hours as you don't need data so that all data will be cleared automatically and then you can set retetion back to 7 days which is default or as per your requirement.    C] You can also create all topics on a new cluster with the required configuration    If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.   Thank you. ... View more

@CDPTester    You can use camel-snmp-kafka-connector to receive traps and poll SNMP (Simple Network Management Protocol) and configure it to send to Kafka topic [1].     [1] https://camel.apache.org/camel-kafka-connector/0.11.x/reference/connectors/camel-snmp-kafka-source-connector.html   This connector will be not bundled with the CDP but you can integrate this Apache open source connector as per your requirement. Refer the following stackoverflow article for more details/example [2]   [2] https://stackoverflow.com/questions/54305856/snmp-camel-kafka   If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.   Thank you. ... View more

@yagoaparecidoti    A] Generally /config/topics/ will store all topic level configuration if you have set any.   For example: If you have set any topic level setting then it will be stored there:   [zk: node1.zookeeper-hostname.com:2181(CONNECTED) 6] get /kafka/config/topics/ATLAS_HOOK {"version":1,"config":{"retention.ms":"2678400000","cleanup.policy":"compact"}} [zk:node1.zookeeper-hostname.com:2181(CONNECTED) 7]   Here topic level settings are retention.ms and cleanup.policy=compact   As it has only topic-level configuration details it will not affect the Kafka cluster or applications if applications are not dependent on any topic-level settings.    B] There is no standard Kafka tool which you can use to restore /config/topics but you can easily alter/set topic level settings using the Kafka command below as per your retirement if any.    kafka-configs --zookeeper <ZK_host>:2181/kafka --entity-type topics --entity-name <topic_name> --alter --add-config <configname & value>   Note: Replace all required settings and /kafka (/kafka can be checked from CM >> Kafka >> Conf >> zookeeper.chroot) in the above command   If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post. ... View more

  @shrikantbm& team, Yes, in this case, we need to check cleanup.policy of the topic __consumer_offsets. If the existing cleanup.policy=compact then the log segment of this topic will not be deleted.   You should follow the below steps to conclude and resolve this issue initially.    1) Check what is current cleanup.policy of the topic __consumer_offsets. You can check it using the command:  kafka-topics.sh --bootstrap-server <broker-hostname:9092> --describe OR kafka-topics.sh --zookeeper <zookeeper-hostname:2181> --describe --topics-with-overrides   Note: topic_name is the name for which you are facing an issue   2)  If you want to clear the old log segment of this topic, then you should set cleanup.policy like cleanup.policy=compact,delete,retention.ms=<30days>    compact = when the kafka-log is rolled over, it will be compacted delete - once the offset.retention.ms is reached, the older logs will be removed retention.ms=<30days>  > the old log segment will be deleted after 30 days.   Note: 30 days are just an example here and this setting will be in ms. You should set it as per your requirement after checking it with the application team and their need.    For "delete", the property "log.cleaner.enable" must be set to "true"   After configuring this cleanup policy data will be deleted as per retention.ms as suggested above. If you will not set retention.ms then old log segment will be deleted as per retention period set in the CM / Ambari >> kafka >> Conf. The setting is log.retention.hours = <7 Days default> in CM >> Kafka, check what it is in your case so that log segment older than 7 days will be deleted. Kafka will keep checking the old log segment with the help of the property log.retention.check.interval.ms .   Important note: The "delete" on consumer offsets is that you may lose offsets which can lead to duplication/data loss. So check it with your application team before setting a deletion policy.   3) If you still face the same issue, then broker logs need to be reviewed for the root cause of the issue and make the changes accordingly.    If you found this information helped with your query, please take a moment to log in and click on KUDOS 🙂 and "Accept as Solution" below this post.   Thank you. ... View more