The Excel files are in a series of different formats - the specific format isn't that important, but each column has a JSON label as the header and one or more rows of data. For example:
The output JSON must match the input file - I can't define a fixed schema as each input file may have different schema.
I have a flow that looks something like this:
(the flow goes on to do other things after adding the submitted_at field).
The ConvertRecord processor is fairly simple and has a CSV reader and a JSON writer. Now, we're seeing the CSV created consistently with US date format (MM/dd/yy) but the API needs it as a UK date format (dd/MM/yyyy).
The CSVReader config:
Note that we're inferring the schema from CSV and defining the date format so that Nifi recognises the format.
For the writer:
This is inheriting the schema - we don't know ahead of time what the schema will be so I can't use a fixed schema. For debugging, I've enabled writing the schema to an attribute so that I could see what it was doing. Note the date format here is in UK format.
When I run this, however, it continues to output the date in US format.
Now, running a test file through with the schema recorded to the avro.schema attribute and defining the schema, the processor worked and converted the date. However, because the input file attributes vary, I can't define a fixed schema and need to infer the schema from the CSV each time.
Am I missing something in how I've configured it? Or have I stumbled accross a defect?