question Re: How to parse w/ fixed width instead of char delimited contents? in Support Questions

How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 00:49:32 GMT

I am trying to parse data from file contents that are generated by fixed width instead of by a delimiter. As a simplified example, the value for data attribute 1 is in position 1-2, for attribute 2 is in position 3-6, and attribute 3 is in position 7-8 in each line. Then, the file contents should be transformed as below.

Before

AABBBBCC

DDEEEEFF

After

AA;BBBB;CC

DD;EEEE;FF

I assume there may be a way to capture substrings per line? Please assist.

Re: How to parse w/ fixed width instead of char delimited contents?

mpayne — Fri, 15 Jan 2016 00:54:51 GMT

Kausha,

You can use ReplaceText to do this. In your example above, you can use a Replacement Strategy of "Regex Replace".

Set Evaluation Mode to "Line-by-Line"

The Search Value would then be:

(.{2})(.{4})(.{2})

And the Replacement Value would be:

$1;$2;$3

Does that help?

Re: How to parse w/ fixed width instead of char delimited contents?

jpercivall — Fri, 15 Jan 2016 00:57:08 GMT

This is a bit tricky and will require a bit of regex magic. You will want to capture each length of data attribute (2, 4 and 2 respectively) into capture groups then use those capture groups to replace the content.

You'll use the ReplaceText processor with a search value of "(.{2})(.{4})(.{2})" and a replacement value of "$1;$2;$3" and configure it to evaluate line by line. This will go through the contents of the flowfile line by line and replace the contents like you want.

Comment below if you run into any problems!

Re: How to parse w/ fixed width instead of char delimited contents?

bsaini — Fri, 15 Jan 2016 01:11:01 GMT

Assuming you are okay with using Hive for this, you would just create a table with one column (column name something like row) and then load the whole file into that table. Run a query to then split the columns and insert in another table.

Here are more details and code snippet. https://martin.atlassian.net/wiki/pages/viewpage.action?pageId=21299205

Re: How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 01:13:01 GMT

Yes. This works well, but is there a way to store the values as attributes. Ultimately, I want to use the AttributesToJSON processor.

Re: How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 01:14:33 GMT

I have posted to @mpayne also: "Yes. This works well, but is there a way to store the values as attributes?" Ultimately, I want to use the AttributesToJSON processor.

Re: How to parse w/ fixed width instead of char delimited contents?

jdyer — Fri, 15 Jan 2016 01:14:56 GMT

I agree with @jpercivall and @mpayne ReplaceText is the best way to go. I created a quick workflow that you can reference. This was assuming the input of AABBBBCC as you suggested. You can change the GetFile path and PutFile path, and the regex in ReplaceText to test with your real data.fixedwidthexample.xml

Re: How to parse w/ fixed width instead of char delimited contents?

bsaini — Fri, 15 Jan 2016 01:16:21 GMT

didnt realize the question was about nifi.. my bad.

Re: How to parse w/ fixed width instead of char delimited contents?

jpercivall — Fri, 15 Jan 2016 01:19:22 GMT

Ah so you also want to extract the text to be attributes of the flowfile. Is the structure of the contents only ever two lines and do you want to create JSON using both of those lines or split them into separate flowfiles?

Re: How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 01:59:36 GMT

The number of rows for the flow files will vary. Each line of data will represent a record/item. Also, I want the data output in its original file, not in separate flowfiles.

Re: How to parse w/ fixed width instead of char delimited contents?

mpayne — Fri, 15 Jan 2016 02:04:37 GMT

You can use the ExtractText processor and provide it a regex also in order to pull the values into attributes. For example, you could have:

field1: (.{2}).{6}

field2: .{2}(.{4}).{2}

field3: .{6}(.{2})

This assumes, though, that each FlowFile has only a single line. You could use SplitText, for example, to split each FlowFile into a separate line perhaps? I think we may need more context about what you're trying to accomplish to provide a more detailed answer.

Re: How to parse w/ fixed width instead of char delimited contents?

jpercivall — Fri, 15 Jan 2016 02:09:02 GMT

What's your end goal JSON schema look like? What identifiers are each of the values going to use (for every line)? As a note, AttributesToJSON only creates flat JSON objects (no nested fields).

Re: How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 02:36:52 GMT

I have assumed the following flow: GetFile --> ExtractText --> SplitText --> UpdateAttribute --> AttributesToJSON --> PutFile

I receive an error in PutFile. Below are my modified configurations

ExtractText - Enable Multiline Mode = True

SplitText - Line Split Count = 1; Header Line Count = 1

Update Attribute - Properties as suggested Att1 = (.{2}).{6}; Att2 = .{2}(.{4}).{2}; Att3 = .{6}(.{2})

AttributesToJSON Attributes List = Att1, Att2, Att3

What am I missing here?

Re: How to parse w/ fixed width instead of char delimited contents?

mpayne — Fri, 15 Jan 2016 04:45:48 GMT

What error do you see in PutFile?

Re: How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 10:07:02 GMT

The output, based on this example would be JSON array

[{"field1":"AA","field2":"BBBB","field3":"CC"},{"field1":"DD","field2":"EEEE","field3":"FF"}]

Re: How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 10:29:56 GMT

I am able to run the flow when I set the ExtractText-->Splittext connection for matched and unmatched, but with incorrect output: {"Att3":" .{6}(.{2})","Att2":".{2}(.{4}).{2}","Att1":"(.{2}).{6}"}.

Would it be more efficient to use the ReplaceTextWithMapping processor? I am unable to find a template with this processor and a relevant mapping file.

Re: How to parse w/ fixed width instead of char delimited contents?

ltsimps1 — Fri, 15 Jan 2016 10:35:20 GMT

I have attempted, unsuccessfully, using the ReplaceText processor. The method works if I have a small/set number of lines in my file.

Do you have any guidance on the ReplaceTextWithMapping processor and how the mapping file should be formatted?

Re: How to parse w/ fixed width instead of char delimited contents?

other — Tue, 09 May 2017 03:08:24 GMT

Hi,

I am not able to replicate same example. I am getting output as $1;$2;$3 all the times. I am new to NiFi and I and not able to find where I am missing. I think I have not used Properties correctly.

Re: How to parse w/ fixed width instead of char delimited contents?

kpulagam — Fri, 28 Oct 2022 21:58:40 GMT

Is it possible to output some part of the input text?

For Example:

Input: AABBBBCC

Output: AA

Re: How to parse w/ fixed width instead of char delimited contents?

Pawa — Thu, 15 Dec 2022 13:24:37 GMT

This is working fine. Can we provide Search Value and Replacement Value as Variable or flowfile attribute. As I wanted to use same replace text processor to convert different input files with different number of columns. Basically I want to parameterised the Search Value and Replacement Value in replace text processor. @mpayne @ltsimps1 @kpulagam @jpercivall @other