Renaming NiFi attributes when you don't know how many attributes could be in the Flowfile at DesignTime

[#1]

Hi All,

Here is some background on what I want to do:

I'm reading HL7 message files and want to create an HBase table with that data.

The ExtractHL7Attributes processor parses HL7 messages and creates attributes in this fashion - <Segment Name> <dot> <Field Index>. If the segment is repeating, the naming will be <Segment Name> <underscore> <Segment Index> <dot> <Field Index>. Examples of attributes it creates are "MHS.12" with a value of "2.1" and "OBX_11.3" with a value of "93000^CPT4".

I want to write these attributes and their values to an HBase table, where the attributes (MHS.12) become column names and the values become the cells in the HBase table. This works fine, but the column names MHS.12 and OBX_11.3 are not very meaningful for an HBase table.

So, using an UpdateAttributes processor, I'm able to rename the columns as follows:

OBX_1.1 --> OBX_1_Set ID

OBX_1.2 --> OBX_1_ValueType

.

.

.

OBX_8.1 --> OBX_8_Set ID

OBX_8.2 --> OBX_8_ValueType

But for this to work, you need to know how many attributes (or OBX segments) will be in the flow file at design time; Since OBX segment can repeat itself many times in the message and since we can’t anticipate the max number of OBX segments a message could have at design time, I am unable to rename all attributes at design time;

Any suggestions on how to make this work ?

Is creating a custom processor the route to take ?

Thanks in advance.

[#2 · Accepted solution]

Raj, I think you can achieve what you're looking to do using an ExecuteScript processor after ExtractHL7Attributes to match against and rename the attributes in question. For example, the Ruby script attached below matches against the segment name and field index, and then uses the `element_names` mapping to rewrite the field index to its respective element name while keeping the segment name and segment index the same.

That said, with as many segments and elements as are in HL7, having to include this mapping may be more cumbersome than you were hoping. You can of course externalize it to a file or DB, but since the HL7 element names are fixed, I don't think the overhead is worth it. I thought a little bit about whether you could use ReplaceTextWithMapping here too but you'll still have to rewrite the attribute names so you're mapping on a contiguous string -- we need to match against segment name and field index, but segment index is in between in the ExtractHL7Attributes output.

[#3]

Thanks very much for the detailed answer, ideas, screenshots, and sample code, that's very helpful.