Hello, We are trying to use the EncryptContent processor to encrypt and decrypt PII data. We are doing Encryption in other applications using JAVA. We are trying to incorporate the same using NiFi. We are able to get EncryptContent working but not able to replicate the exact logic that we use in other applications. We need to be in sync as we share the data. The way, we are doing it in other applications is, we use the AES-CBC-PKC5Padding Keyed encryption along with a Initialization Vector and the Secret Key used, is encrypted using Standard PBE. I would like to know how to replicate this in NiFi. How do I pass the Initialization Vector? Please help. NOTE : I have this working as ExecuteScript and I am also able to use EncryptContent using the raw key hex value without KDF. Thanks, Prakash ... View more

As part of our application, we are consuming events from Kafka and we are doing data transformation before sending the events to downstream system. When we are running load as part of Performance testing, we noticed we are not getting even distribution across all the nodes through ConsumeKafka. We have 9 nodes in the cluster. I have attached the document that has snapshots of Kafka properties, count of events on all nodes from Splunk and Flow file status history from ConsumeKafka in NiFi. Trying to understand the behaviour here. And we have also noticed the below error intermittently o.a.n.p.kafka.pubsub.ConsumeKafka org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed due to group rebalance 1) Why is the data not distributed across the nodes? 2) Why do we see the error intermittently? ... View more

I havent gone through the code for the DistributedCache yet. After the first try today, I had a few questions that came up. 1) Do we have to use different Distributed cache Server/Client for each Processor that has State management? For example can we use the same Distributed Cache Serve/Client for both ListSFTP and ListHDFS within the Processor Group? 2) If we specify a value in the "persistence directory" in the DistributedCacheServer, the assumption is the cache is present in both the memory and in Disk? Is the understanding correct? 3) The expectation of using the DistributedCachingService is that the state is maintained across the cluster which means even if we lose the node that was primarily running the Processor, we still do not duplicate the listing. But when I try to understand the thread in the link http://mail-archives.apache.org/mod_mbox/nifi-users/201611.mbox/%3CCA%2BWJ-%2B%2Bqdkg-qRzP-7gUAX%2BAsjj%2Bm%2BH9zoF9LkjgriHKDMgPXw%40mail.gmail.com%3E specifically the line "it does not implement any coordination logic that work nicely with NiFi cluster" I am not sure I exactly follow the issue. Please clarify. ... View more