@Matt Clarke Thanks your reply, I did follow the second option. But I had to remove data content on the disconnected node before restarting it. And I found that the node disconnected because of overload queue when executing job. I confuse that if we can configure queue size up to contain more data. How can we do this? Please help me if you have solutions for these problems. (overload queue). Thanks, Kiem ... View more

@Wynner Yes. I think problem is about configure number of concurrent task on Consumer_Kafka_0_10 processors. I will accept your answer as you determine right problem. But Can you help me configure number of concurrent task with my case: In my case, I have 3 PublishKafka_0_10 processors A, B, C. A push data to topic aa, B to topic bb and C to topic cc. Each PublishKafka_0_10 processor has number of concurrent task is default 1. As you see, my Kafka cluster has 4 partitions for all 3 topics. Then I have 3 Consumer_Kafka_0_10 processors D, E, F. D receives data from topic aa, E receives data from topic bb, F receives data from topic cc. How many concurrent task need to configure for each Consumer_Kafka_0_10 processors D, E, F? Please help me understand relation between partition and concurrent task. Thank you so much! ... View more

@Wynnwe I have 3 topics in Kafka cluster: aa, bb, cc. I use this syntax to check number of partitions. It seems have 4 partition. ./bin/kafka-topics.sh --describe --zookeeper 10.42.53.16:2181,10.42.53.17:2181,10.42.53.18:2181 --topic aa Result: Topic:aa PartitionCount:4 ReplicationFactor:1 Configs: Topic: aa Partition: 0 Leader: 17 Replicas: 17 Isr: 17 Topic: aa Partition: 1 Leader: 18 Replicas: 18 Isr: 18 Topic: aa Partition: 2 Leader: 17 Replicas: 17 Isr: 17 Topic: aa Partition: 3 Leader: 18 Replicas: 18 Isr: 18 It is same result when I check with topic bb and cc. However, the loss of the package just occurs in ConsumerKafka_0_10 processor that receives data from topic aa or bb. The ConsumerKafka_0_10 processor that receives data from topic cc is always enough. ... View more

@Kiran Hebbar Hi Kiran Hebbar, I think you can use StreamCallback to read your content from a Flowfile. You can modify or add more attributes into flowfile. Here I find a Matt Burgess's example using Javascript: http://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited.html Hope that help you, Thanks, ... View more

It works well. Thanks Matt !!! Hadoop Configuration Resources Properties: point to core-site.xml and hdfs-site.xml after copying them into each Nifi node. ... View more

Thanks @Pierre Villard, That exactly what I want: use a RPG pointing to the cluster itself. ... View more