Member since
03-22-2019
9
Posts
0
Kudos Received
0
Solutions
04-26-2019
08:36 AM
@Shu This solution doesn't work if I have multiple arrays in one json message. Example - {"TxnMessage": {"HeaderData": {"EventCatg": "F"},"PostInrlTxn": {"Key": {"Acnt": "1234567890","Date": "20181018"},"Id": "3456","AdDa": {"Area": [{"HgmId": "","HntAm": 0},{"HgmId": "","HntAm": 0}]},"escTx": "Reload","seTb": {"seEnt": [{"seId": "CAKE","rCd": 678},{"seId": "","rCd": 0}]},"Bal": 6766}}} If you could help with analysis and/or pointers on how each element of array in such input can be split into a separate rows?
... View more
04-04-2019
11:15 AM
@Matt Clarke Thanks a lot. You are genius!
... View more
03-29-2019
06:58 PM
@Matt Clarke Its a nifi cluster having 3 nodes. That's why I am getting 3 files. Is there any way to handle this? I have already I invested my 4 days into this. Lol.
... View more
03-29-2019
09:55 AM
@Matt Clarke I have increased the Run Schedule to 2 mins and made sure that queue has all the flowfiles before executing MergeRecord processor. Still my output is 3 files of 4KB each but the desired outcome is one merged file. Not sure what configuration is impacting the result.
... View more
03-28-2019
01:43 PM
We are receiving Json messages from upstream system via Kafka topic. Requirement is to store these messages into HDFS at certain interval. Since we are storing into HDFS we want to merge certain number of these Records in to single file. As per NiFi documentation we are using "MergeRecords" processor for that. About the in coming Records:## These are the multi-line JSon messages with nested structure. Those are based on the same schema (they are picked from single Kafka topic) Those are validated messages and even NiFi processor is able to parse it. so apparently no issues with JSon messages from Schema point of view Present Configuration Below is the snapshot of the Processor Configuration. NiFi version: 1.8 Expected behavior For the Above configuration its expected that MergeRecords should have weighted for one of the thresholds i.e. Maximum records(100000) or Maximum Bean size(100KBs). Observed Behavior But its observed that bean is getting bundled pretty before either of the threshold is reached. It is triggering the bean formation only for 2 records of 5KB size. If you could help with analysis and/or pointers as why MergeRecord processor is not behaving as per the configuration? @Matt Burgess @Matt Clarke
... View more
Labels:
- Labels:
-
Apache NiFi