Nifi: Read and convert with custom Schema csv with binary delimiter

AndreyDE — Tue, 19 Nov 2024 09:31:24 GMT

Hello!

The source csv file is:

123456TextValue1
654321TextValue2

where 123456 and TextValue1 are different values, separated by binary delimiter (\u0001)

Similar, 654321 and TextValue2 have non-visible at Web-UI binary delimiter (\u0001)

I use ConvertRecord for updating the delimiter from "\u0001" to ";"

RecordReader is CSVReader with the following properties:

- Schema Access Strategy: Use 'Schema Text' Property
- Schema Text: #{test_schema}
- Value Separator: \u0001
- Treat First Line as Header: false
- Ignore CSV Header Column Names: true

RecordWriter is CSVRecordSetWriter:

- Schema Access Strategy: Use 'Schema Text' Property
- Schema Text: #{test_schema}
- Value Separator: ;
- Include Header Line: true

test_schema is

{
"type": "record",
"name": "test_schema",
"fields": [
{
"name": "FIELD_1",
"type": ["int","null"],
"description": "FIELD_1"
},
{
"name": "FIELD_2",
"type": ["string","null"],
"description": "FIELD_2"
}
]
}

Expected output is:

FIELD_1;FIELD_2;
123456;TextValue1
654321;TextValue2

But I got the following error:

ERROR
ConvertRecord[id=01931001-0d7e-1e43-146d-1a380e6d43b7] Failed to process FlowFile[filename=7365b509-7100-4bc2-a070- 4cc8ce8377b9]; will route to failure: org.apache.nifi.processor.exception.ProcessException: Could not parse incoming data
Caused by: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record
Caused by: java.lang.NumberFormatException: For input string: "123456TextValue1"

Re: Nifi: Read and convert with custom Schema csv with binary delimiter

satz — Tue, 19 Nov 2024 10:55:41 GMT

@AndreyDE Thank you for posting your query.

As per the CSVReader docs:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.27.0/org.apache.nifi.csv.CSVReader/index.html

The property Value Separator only considered if the [CSV Format] Property has a value of "Custom Format".

Does your CSV Format is set with "Custom Format" ?

Re: Nifi: Read and convert with custom Schema csv with binary delimiter

AndreyDE — Tue, 19 Nov 2024 12:41:16 GMT

Yes, CSV Format is Custom Format

question Re: Nifi: Read and convert with custom Schema csv with binary delimiter in Support Questions

Nifi: Read and convert with custom Schema csv with binary delimiter

Re: Nifi: Read and convert with custom Schema csv with binary delimiter

Re: Nifi: Read and convert with custom Schema csv with binary delimiter