Iam new to Alerting & Monitoring. If we want to setup alerts for nifi using splunk can we use putsplunk nifi processor or send log files directly to splunk? Currently we are having applications use splunk where they send the log files directly to splunk for alerting. Which is the effective way to acheive monitoring and alerting for nifi using splunk? Thank you ... View more

What is the best practice to productionize nifi and schedule it to run every hour through a script? How do we upgrade nifi to a new version. How to deploy new workflows and modified workflows into production in EC2. How do we deploy changes into nifi workflows thank you ... View more

I think concurrenttasks is having 12 threads for this getkafka processor i.e. 12 consumers in one consumer group? Please correct me if iam wrong. So if i have 12 partitions in my kafka topic then i believe 12 consumers are consuming from 12 partitions? I have a 3 node nifi cluster and made concurrent tasks to 4 so that i can split the load between all the nifi nodes in the cluster. I think each node will be consuming from 4 paritions. What happens to the 4 partitions of that node if it dies or crashes? Normally there will be a rebalance and kafka will reassign partitions to consumer so that the 2 node will consume from 6 partitions each? Is this how it works in Nifi? What will happen to the messages or offsets that has already in the queue of a node that has died? Is there anyway to store offsets in nifi to achieve fault tolerance? ... View more

I mean in which logs the nifi warnings and errors are stored? ... View more

The current workflow is exporting each event. We are looking to merge all json events based on service/eventname and concatenate time and export them to s3. Our requirement on and merge them using expression language at the runtime. ... View more

Currently we are looking to use Nifi to get Json events from kafka and store them to s3. We are looking to set up a nifi cluster and i have following questions please? 1. What happens to Nifi when I do a rolling restart of kafka? I think nifi listens to zookeeper :2181 or kafka nodes how do we configure it in GetKafka processor? 2. What does Nifi do if the peered connection becomes unavailable? 3. How does it handle a kafka node going offline? 4. Does it always connect to the same kafka node or what determines which node to connect to? 5. Does it connect on IP or DNS (kafka IP's can change)? Can we make Nifi to connect to kafka DNS rather than IP. 6. What will it do if a leadership election is invoked while its consuming? 7. What monitoring and alerting will be in place in production? 8. Any issues if Zookeeper embedded in slaves and how Zookeeper maintains the state integrity if we have more than 1 zookeeper in the nifi cluster? Or is it better to have a separate single ZK instance in cluster? thanks ... View more