Support Questions
Find answers, ask questions, and share your expertise

Flume in Flafka lag issue

Flume in Flafka lag issue

New Contributor

We have a Flafka cluster with Kafka-Ambari-Zookeeper-Flume-RabbitMQ-HDFS Clouderra stack, we had a electric surge a month ago, after that incident, we lost some blocks in HDF with bad, but now Flume fails to deliver the same count of topic partition messages to HDFS, there is no lag or offset difference, but balance of messages flown from Kafka is dissipated in Flume sinks or lost without reaching HDFS destination successfully, flume dissolving messages in the headband. We are having problems sinking hdfs. When it restarts, it corrects, but after a certain time flume cannot write.