NiFi: Convert a proprietary ASCII based format to CSV

SergiyK — Wed, 15 Apr 2020 12:32:34 GMT

My data is coming in text (ASCII) files, with each line having a fixed number of fields and each field having a fixed length. I want to convert this files to CSV, like this:

Input line:

abbccc

Output line:

a,bb,ccc

I the example above "a", "bb", "ccc" are fields of fixed length. So, I always know exactly how to split the input line.

I looked into the ConvertRecord operator and the ScriptedReader controller service (that can be used as a record reader), but I was not able to find any example of a Python script for ScriptedReader. I found this ExecuteScript Cookbook by @mburgess, but those recipes are much more general, so I cannot use them in SciptedReader (which needs very specific objects for record processing that must be created by the script).

Can anyone give a basic example of a Python script that can be used in ScriptedReader to process records?

Alternatively, is there another way to accomplish the task (another processor)? Of course, I can use ExecuteScript processor and script the processing of complete FlowFiles in it, but my FlowFiles contain millions of records and I think this processing will be much more inefficient than SciptedReader.

Re: NiFi: Convert a proprietary ASCII based format to CSV

aakulov — Wed, 15 Apr 2020 15:16:34 GMT

You can try the ReplaceText NiFi processor withe the approached described here. That will be a clean way of doing what you want without much scripting.

question Re: NiFi: Convert a proprietary ASCII based format to CSV in Support Questions

NiFi: Convert a proprietary ASCII based format to CSV

Re: NiFi: Convert a proprietary ASCII based format to CSV