We have multiple nifi users and separate dev, test, and prod nifi instances. Our users are developing flows in dev and moving them to test and prod via a central flow registry instance. If we upgrade the lower environments before the higher, a user can develop flows that will not instantiate on the higher env due to them being developed with a higher version of nifi. But we want to upgrade the lower envs first to allow issues to surface in dev and test before we upgrade prod. My questions are What are the actual between-version flow compatibility goals of nifi? Are we guaranteed backward compatibility but not forward? What is a good strategy for handling nifi upgrades in a traditional dev->test->prod configuration, while maintaing availability and enabling mostly painless SDLC? ... View more

We have a need to archive provenance data indefinitely. a simple file based archival to hdfs would meet our needs. From reading questions on this site, I know we can use the nifi site-to-site reporting task to send provenance events as flowfiles via nifif site-to-site. The obvious but probably wrong solution would be to point the reporting task to its own nifi cluster, catch the flowfiles and do a mergcontent -> puthdfs. This is probably wrong because this flow itself would generate more provenance events.. which would generate more provenance events.... forever. I'd really like to avoid the administrative burden of running another nifi instance, even a minifi instance. Has anyone come up with a good solution for archiving provenance without using another nifi cluster? ... View more

I am having a hard time understanding how to build a flow for a simple use case. The use case is doing an incremental fetch from a rest api. I just need to store the last index I have retrieved, and use that index as the starting index for my next fetch. I can think of a few ways of doing this but all of them seem to have problems. Here are my ideas and thoughts, how are other people tackling this use case? UpdateAttribute Stored State Use updateAttribute with stored state to store my index. Problem: update attribute only stores state locally, if my node goes down or there is a primary node switch, thats a problem Store State in Flowfiles Loop the output of my getHTTP through some updateAttribute stages that do something like ${new_beginning}=${last_end} Problem: the node with my state-storing flowfiles could go down. No way to "drain" a node if its got these long-lived flowfiles Store State in an external RDBMS Just store my index in some external db Problem: none really, just extra burden and dependencies Distributed Cache Controllers Talk to a controller service that stores state. Problem: From researching I find out that distributedMapCache isn't actually cluster wide, do I understand that correctly? Am I missing anything? How are others solving this use case that I imagine is very common? ... View more

I have csv files that are delimited with ascii 01 bytes. I want to utilize a CSVReader controller to read these records and convert to other formats. My problem is that I don't know how to input a 0x01 byte into the CSVReader "Value Separator" property. Is there a standard way to do hex escapes in nifi properties? This property does not support the nifi expression language. Thanks! ... View more