Ah, I somehow thought you were on 1.1 rather than 1.10.   In case you are on 1.10: Please check the documentation of your relevant distribution. As far as I know the latest versions of our platform are working with Nifi 1.9 (not yet 1.10).   If you somehow found a newer version of Nifi, it is possible you got the raw Apache version, rather than the Cloudera improved version which has fixes in numerous areas (and presumably with the integration of the version registry as well). ... View more

I just tested the following with Nifi 1.9   1. Create variable 2. Use in processor, start processor 3. Copy processor 4. Update variable 5. Copy entire process group 6. Update variable   This flow went through smooth.   I think the main concern here is that you are running Nifi 8 versions behind the latest, and my recommendation would be to update to the latest one.    ----   I did not find anything about this specific case, but this very similar issue was solved in 1.6, providing good confidence that using a recent version is the recommended course of action: https://jira.apache.org/jira/browse/NIFI-5033 ... View more

It seems that the PutDatabaseRecord processor expects your timestamp to be numeric (probably milliseconds since the epoch).   I notice your screenshot contains a specification of the timestamp format, but I don't see this as an option in the documentation, so possibly that has no effect in this processor.   Here is the reference: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.PutDatabaseRecord/index.html   ---- Possible solution: Load or convert the timestamp to the right value.   A simple trick to check what the value should be, is by manually inserting a date into the table, loading it with Nifi and then inspecting the value.   (You can also test writing it back, that should confirm that the value works).   ... View more

This is slightly confusing, but if you have a Mysql query to generate the difference, then Nifi can get the data from there and convert to XML if needed.    As generating the difference seems to be the main challenge here, perhaps start by finding a way to do this, and from here it should be trivial to design the Nifi flow around this.   ---- If this answers the question, consider marking this as the answer  ... View more

Please confirm that you are trying to solve a real problem.   The fact that you need to wait for files to be completely available is not that specific to huge files, also I have not run into this with Nifi specifically. Most filesystems should handle this in a good way automatically.    A normal thing to see is that a file is either hidden, or in a different location until it has become complete.    ---- If this answers the questions, consider marking this as the answer.      ... View more

Sending an email every time an error occurs may be problatic. The only thing that seems to come close is the site to site bulletin report Processor.   As mentioned you can find the messages in the logs. If your main concern is accessibility, look into LogSearch on HDP.  ... View more

I hope you just want to do this once, in that case you can just write a query that allows you to get all records later than a certain timestamp.   If you want do do this continuously or there is no query that gives the new records, you are looking for Change Data Capture. I have seen a simple processor for Mysql, but in general you may want a dedicated CDC tool to help Nifi pick up the changes. ... View more

Perhaps you are looking for something simple, but comparing XML can be quite a complex topic in general. Therefore I don't expect this to become available in the form of a standard processor.    I would recommend looking into a python script to do this, and then calling this from Nifi.  ... View more

A quick search suggests that libhdfs can do it, but I have not tried it myself. ... View more