question Re: NiFi convert string to key/value pair in Support Questions

NiFi convert string to key/value pair

hans_feldmann — Fri, 22 Jul 2016 23:10:18 GMT

i'm still learning the ropes of NiFi, I have some XML (See Below)

<measInfo measInfoId="1542455297">
	<measTypes>1542455297 1542455298 1542455299 1542455300 1542455301 1542455302 1542455303 1542455304 1542455305 1542455306 1542455307 1542460296 1542460297 </measTypes>
	<measValue measObjLdn="LTHAB0113422/ETHPORT:Cabinet No.=0, Subrack No.=1, Slot No.=7, Port No.=0, Subboard Type=BASE_BOARD">
		<measResults>116967973 585560 496041572 682500 0 12583680 72080 520454 46670568 73432 2205837 1000000 1000000 </measResults>
	</measValue>
	<measValue measObjLdn="LTHAB0113422/ETHPORT:Cabinet No.=0, Subrack No.=1, Slot No.=7, Port No.=1, Subboard Type=BASE_BOARD">
		<measResults>0 0 0 0 0 0 0 0 0 0 0 0 0 </measResults>
	</measValue>
</measInfo>

I successfully parsed the XML, but need further processing. I'm looking for suggestions as to the best way to handle this. Ultimately I would like a key/value pair between MeasTypes and MeasResults to insert into Hbase or Pivot data into a Hive table (MeasType would get translated into a Column Name). The exact number of MeasTypes and MeasResults pairs is variable.

measTypes	measResults
1542455297	116967973
1542455298	585560
1542455299	496041572
1542455300	682500
1542455301	0
1542455302	12583680
1542455303	72080
1542455304	520454
1542455305	46670568
1542455306	73432
1542455307	2205837
1542460296	1000000
1542460297	1000000

I have 10GB of XML arriving every 30 minutes, so efficiency is also a concern. I'm able to take the measTypes and put it into a file and parse it with a ExtractText and ReplaceText (using a Regex), so the next challenge is to replace the GetFIle with a feed from my existing XML template, how do I do it? Another Option is to InvokeScriptedProcessor (Maybe Python)? Sample code would be greatly appreciated. 🙂

Re: NiFi convert string to key/value pair

mpandit — Mon, 25 Jul 2016 20:17:09 GMT

One option I can think of is using xquery or xslt.

Re: NiFi convert string to key/value pair

TimothySpann — Mon, 25 Jul 2016 20:35:20 GMT

Here are some NiFi XSLT references

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.TransformXml/

http://stackoverflow.com/questions/6104698/use-xsl-to-convert-an-xml-file-to-name-value-pair

Re: NiFi convert string to key/value pair

hans_feldmann — Mon, 25 Jul 2016 21:48:11 GMT

Wow, didn't know the xQuery could do this. Based on quick research it appears that fn:tokenize might be just what I'm looking for. I'll have to play with this to see if I can this to work.

Re: NiFi convert string to key/value pair

rbnaveen_kumar — Tue, 10 Apr 2018 21:04:40 GMT

Hi Hans

Did you manage to accomplish this using xquery? Your solution would be greatly appreciated. Thanks

Re: NiFi convert string to key/value pair

hans_feldmann — Wed, 11 Apr 2018 01:49:50 GMT

Sorry I did not get this to work in xQuery. I ended up using XSLT to do the transformation.