You can use the SiteToSiteProvenanceReportingTask for this. Filter the reporting task to only emit events at the "certain point" you mention above. Each event has a "timestampMillis" and "lineageStart" field, you should be able to route on the difference of the two using QueryRecord, with something like: SELECT * FROM FLOWFILE WHERE timestampMillis - lineageStart > 60000 Which should emit a flowfile containing all events for which the associated entity (in this case, the flow file in the system) has been in the flow for over a minute. ... View more

You'll need to provide the CLOB as an attribute, meaning you've set attributes like sql.args.1.type to 2005 and sql.args.1.value to the CLOB value. Then your SQL statement would have a ? parameter, and the CLOB value will be inserted when the SQL statement is prepared. See NIFI-4352 for more information. ... View more

The hard part here is that Hive returns STRUCT columns as JSON strings, so even if we can parse the JSON, we've lost the type information. It's possible we can retrieve it from the metadata and (if so) create a nested record from the results. Please feel free to file a Jira for this enhancement. ... View more

The JOLT DSL takes a lot of practice, but once it clicks, it's like seeing the Matrix lol. I'm no expert but I've put my time in 😉 ... View more

Each flow file in the system automatically has an attribute "uuid" set to a unique identifier, this attribute is available in provenance events, via Expression Language (use "uuid" instead of "UUID()"). Is this what you are looking for? Or do you want a separate attribute that would remain the same even if the flow changes flow files (Split, Merge, e.g.)? ... View more

1) I believe TransformXML is more flexible in terms of structural transformation as it leverages the full power of XSLT. However the XML-to-JSON XSLTs I've seen sometimes have limitations (inline comments can be a problem, e.g.). 2) I'm not sure which would be faster per se, probably depends on how much data, what kind of transformation(s) are performed, etc. Also I think TransformXML reads the entire XML input into memory, so for large XML files you may risk running out of memory. ConvertRecord's record readers read in a single record at a time IIRC. 3) It doesn't seem like you want to convert anything to Avro, are you asking how to see the record schema? Internally we have our own RecordSchema representation, but when we write out to a flow file attribute (for example), we use Avro's schema format (even if the data is in CSV, JSON, XML, etc.). To see the schema, set your RecordSetWriter to write the schema to the avro.schema attribute, then you can inspect the flow file's attributes from the UI and see the Avro schema. 4) ConvertRecord only changes the format of the input (CSV to JSON, e.g.), it doesn't really do any transformation of the records (although technically you can configure it to add or remove fields). If you're doing any actual transformation of data (uppercasing field names, changing "F" to "Female", etc.) then you can use JoltTransformRecord, UpdateRecord, etc. The key is that all record-based processors will do format conversion for you, so you only need ConvertRecord if all you want is to change the format of the data. Otherwise the other record processors do their thing (like PartitionRecord groups records by value) but will also convert the format, depending on which Reader and Writer you configure. Does that make sense? ... View more

This is an open issue (NIFI-4957), I pinged the original author/contributor to get the status of it, if he is not actively working it I can take a look at adding the capability. ... View more