Resolving Incorrect Fragment Count Values When Splitting and Merging FlowFiles in Apache NiFi

Question

I have a flow in NiFi which splits a file into individual lines, inserts those lines into a database and after those have been inserted updates a control table. The control table only updates after every line has been inserted. To achieve this, the fragment.index is compared to fragment.count - if these are equal, then I know that every line has been processed and we can move on to updating the control table.

However recently some of our files failed to update the control table. I have outputted the Attributes of the flow files to disk, and it shows something that confuses me: the number of flow files that comes out of the split text processor is 66430, which matches the number of lines in the file. However, the fragment.count attribute is 66443.

Does anybody know why the fragment index would be incorrect, and how I can fix this?

MattWho · Accepted Answer

@Mark HeydenrychThe default configuration of the SplitText processor is to not emit FlowFiles where the content is just a blank line.  This behavior is controlled by the "Remove trailing Newlines" property.  The fragment.count attributes is set based on the total number of fragments in the original FlowFile's content.  The Fragment.index is is a one up number assigned to each FlowFile emitted.  So in your case, i suspect that your original FlowFile's content contained 66,443 lines with 13 of those lines as just blank lines that were not emitted.If you change "Remove trailing Newlines" to "false", your emitted count will match your Fragment.count.  Thanks,Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.