Member since
12-05-2016
25
Posts
1
Kudos Received
0
Solutions
01-27-2017
01:48 AM
I reran with the LogAttribute processor, but did not see any flow files going into it. screen-shot-2017-01-26-at-84457-pm.png
... View more
01-27-2017
01:46 AM
I don't see anything flowing into LogAttribute - snapshot of flow attached.screen-shot-2017-01-26-at-84457-pm.png
... View more
01-26-2017
03:28 PM
GM - The template is attached. Appreciate your help!hcc-mergecontent-issue-support.xml
... View more
01-26-2017
03:50 AM
I am unable to set blank for max since MergeContent complains it is not a valid integer. Also set min and max to the same value 325070 as well as 1/325070 but still get the fragment error.
... View more
01-25-2017
05:41 PM
Hello Bryan - I changed to use 1 bin, 0 sec run duration and 1 min/ no max and now get the fragment error once again. Also reset the backpressure object threshold back to 10000. Not sure what i am missing.screen-shot-2017-01-25-at-123644-pm.png "o.a.n.processors.standard.MergeContent MergeContent[id=a8a50e76-0159-1000-371d-9097f2bda225] Cannot Defragment FlowFiles with Fragment Identifier bd01dfda-a8f5-4d37-a587-f7fe853fed3f because the expected number of fragments is 325070 but found only 100 fragments; routing 100 FlowFiles to failure"screen-shot-2017-01-25-at-123911-pm.png
... View more
01-25-2017
02:55 PM
An update - with this configuration, my workflow did not complete even 7 hours later, but MergeContent had not kicked off yet either so no fragment errors. Appreciate any suggestions.
... View more
01-25-2017
04:51 AM
mergecontentissue-template.pngmergecontentconfig.png My sql query returns about 325K records, so in my latest run set my min entries to that(could not use ${executesql.row.count} - MergeContent complained that it wasn't an integer) and max to an arbitrary 500K with 100 bins. I set run schedule as 60 minutes and on the incoming connect I set back pressure number of objects to 400K and back pressure data size to 1GB. The fragment.identifier is the same across the flow files and the fragment.index does look different and less than 325k for some files i sampled in provenance history. I am currently running this configuration, but wanted to share to get your feedback.
... View more
01-24-2017
09:47 PM
I now see the fragment.* attributes in SplitAvro on upgrading to 1.1.1 but still have the issue with x number of fragments expected and y found where x>y in MergeContent. The fragment.identifier and index for a sampling of files is unique. Wondering again if this is back pressure related. Appreciate any help, please let me know if i should open a different thread.
... View more
01-23-2017
08:33 PM
Hello, I am using Nifi 1.0.0 and am trying to merge records from an ExecuteSql processor using MergeContent. I wanted to try Defrag merge strategy and have the following setup in an upstream UpdateAttribute processor for each flow file: 1. fragment.identifier - mmddyy of the flow file 2. fragment.index - nextInt() 3. fragment.count - executesql.row.count 4. segment.original.filename - filename When i run the workflow - i get this error : Cannot Defragment FlowFiles with Fragment Identifier because the expected number of fragments is <sql record count> but found only 1 fragments. It seems like MergeContent is trying to merge too soon - appreciate any advice. My workflow is ExecuteSql -> SplitAvro -> UpdateAttribute (adds fragment.* attribute - could not see these on SplitAvro even though doc indicates it should be present) -> ConvertAvroToJson -> EvaluateJsonPath (to extract only some sql columns) -> ReplaceText(for conversion to comma delimited) -> MergeContent -> PutFile NOTE: I got inconsistent file lengths when trying out various MergeContent Bin-packing configurations so turned to Defrag. thanks
... View more
Labels:
- Labels:
-
Apache NiFi