Support Questions

Find answers, ask questions, and share your expertise

JsonRecordSetWriter not using date format field on inherited schema

New Contributor

I'm currently trying to put together a flow which takes a series of Excel spreadsheets containing records (one per line) and converting them to JSON for use with an API. I'm having some problems with UK/US date formatting and despite following the JsonRecordSetWriter appears to ignore the date formatting.


The Excel files are in a series of different formats - the specific format isn't that important, but each column has a JSON label as the header and one or more rows of data. For example:



The output JSON must match the input file - I can't define a fixed schema as each input file may have different schema.


I have a flow that looks something like this:


(the flow goes on to do other things after adding the submitted_at field).


The ConvertRecord processor is fairly simple and has a CSV reader and a JSON writer. Now, we're seeing the CSV created consistently with US date format (MM/dd/yy) but the API needs it as a UK date format (dd/MM/yyyy).


The CSVReader config:



Note that we're inferring the schema from CSV and defining the date format so that Nifi recognises the format.


For the writer:



This is inheriting the schema - we don't know ahead of time what the schema will be so I can't use a fixed schema. For debugging, I've enabled writing the schema to an attribute so that I could see what it was doing. Note the date format here is in UK format.


When I run this, however, it continues to output the date in US format. 






The generated schema suggess that it's picking up the date format correctly:





Now, running a test file through with the schema recorded to the avro.schema attribute and defining the schema, the processor worked and converted the date. However, because the input file attributes vary, I can't define a fixed schema and need to infer the schema from the CSV each time.


Am I missing something in how I've configured it? Or have I stumbled accross a defect?




Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.