I understand this was hard to find.   One point that is not relevant now, but may help with the general understanding:   If you make a typo in the field names, avro will just ignore the actions of the right field and instead process the new field, which will be empty.    This is technically quite a different problem than when you actually make a typo in the Type, as those are possible to check against the list of possible types.   So detection of typos in field names is hard, and despite what the name suggests it seems LogicalType is actually a field name from Nifi perspective.  ... View more

I see, as there are several steps here it is a bit hard to pinpoint the problem. Please check what the timestamp looks like just before you write it to the database. If it still looks like a string, presumably something went wrong with the reading. ... View more

Ah, I somehow thought you were on 1.1 rather than 1.10.   In case you are on 1.10: Please check the documentation of your relevant distribution. As far as I know the latest versions of our platform are working with Nifi 1.9 (not yet 1.10).   If you somehow found a newer version of Nifi, it is possible you got the raw Apache version, rather than the Cloudera improved version which has fixes in numerous areas (and presumably with the integration of the version registry as well). ... View more

I just tested the following with Nifi 1.9   1. Create variable 2. Use in processor, start processor 3. Copy processor 4. Update variable 5. Copy entire process group 6. Update variable   This flow went through smooth.   I think the main concern here is that you are running Nifi 8 versions behind the latest, and my recommendation would be to update to the latest one.    ----   I did not find anything about this specific case, but this very similar issue was solved in 1.6, providing good confidence that using a recent version is the recommended course of action: https://jira.apache.org/jira/browse/NIFI-5033 ... View more

It seems that the PutDatabaseRecord processor expects your timestamp to be numeric (probably milliseconds since the epoch).   I notice your screenshot contains a specification of the timestamp format, but I don't see this as an option in the documentation, so possibly that has no effect in this processor.   Here is the reference: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.PutDatabaseRecord/index.html   ---- Possible solution: Load or convert the timestamp to the right value.   A simple trick to check what the value should be, is by manually inserting a date into the table, loading it with Nifi and then inspecting the value.   (You can also test writing it back, that should confirm that the value works).   ... View more

A quick search suggests that libhdfs can do it, but I have not tried it myself. ... View more

If you are putting the data in HDFS first, I assume the following python script is more batch than streaming. In that case, consider running it via a scheduler like Oozie. Also, if you run into scalability issues with your script, consider using something like pyspark instead. ... View more

I just checked with @MattWho and it seems this explanation is still relevant. Of course it does assume you are using Ambari:   https://community.cloudera.com/t5/Community-Articles/HDF-2-x-Adding-a-new-NiFi-Node-to-an-existing-secured-NiFi/ta-p/249284 ... View more