Hi All, I got an error while using the ConsumeKafka_0_10 processor . Below is the complete error from nifi-app.log 2018-02-16 11:47:11,465 WARN [Timer-Driven Process Thread-2] o.a.n.p.kafka.pubsub.ConsumeKafka_0_10 ConsumeKafka_0_10[id=b846165f-115a-1161-cadd-815bbb3a45dd] Was interrupted while trying to communicate with Kafka with lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@6cb8afba. Will roll back session and discard any partially received data. bootstrap.servers = [xxx-xxxkafka-xxxz.xxx.xx.local:9093] key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer The last time I saw this error I restarted the nifi and it was gone but it did not work this time. There hasn't been any change on the kafka side. I'm using nifi 1.4.0 along with kafka_2.11-0.11.0.1 to consume kafka records , please let me know if any one knows the root cause of this error. ... View more

I am working on building a monitoring solution for my Nifi workflow (Real time data lake using Golden-gate/ Kafka). Currently I am storing all the Goldgen gate records/flowfiles received from kafka in to a hdfs directory and at the end of the workflow ,in case its ingested successfully in hbase the flowfile is deleted from the directory. So i know that the json format flowfiles left in the Hdfs directory are ones that have failed. Now the issue with finding the reason of the failure is that Nifi's bulletin board shows only the record for last 5/10 mins (not sure of the duration) . I've tried storing the bulletin board messages in hdfs using the REST API but the Json generated from there is very detailed and would require a lot of work before I can use it for monitoring purpose. Has anyone else worked with such type of monitoring? I would also like to know the throughput of the workflow which would include the no of records failed or successfully ingested etc. I know i can get the last-5-min stat from the status history. But if any one else has worked on a similar monitoring task kindly let me know. ... View more