@Shu Great description and thanks for the example. It works just the way I need! ... View more

Hi @Artjom Zab ! So a fundamental difference in error handling is whether the flowfiles have to processed again after error correction or not. OK but beside of this aspect one has always to find the appropriate error information depending on where the error occurs. My aim is still to have a centralized processgroup for uniform error-handling which should log the information to a DB-table to point the support into right direction like the cutout of my flow shows. So I have to verify where a flowfile comes from. IMHO there are three different situations: 1. Processors like InvokeHTTP which deliver suitable information about an error as flowfile attribute OK possible to handle. 2. Processors which send the flowfile to failure-queue WITHOUT setting flowfile attributes or creating entries in bulletin-board, e.g. SplitJson, JoltTransformJSON I don't believe that it is wise to set an UpdateAttribute-processor after every failure-queue and hardcode the component information to attributes. If I look at the data provenance I see componentID and componentName in the details but I don't see them at flowfile in failure-queue. Why...? 3. Processors which generate an entry in bulletin-board, e.g. InvokeHTTP on time-out, ExecuteSQL on time-out, and sending the flowfile to failure-queue Thanks for the link for configurate "REPORTING TASKS". Tested it on my local installation (no permission on the customer DEV-System). But at the moment I don't see how to bed it in my flows. I think this would be a parallel action to monitor this and not being part of my flows. If I would always know the componentID I assume I could handle ALL these cases with the REST-API (e.g. http://127.0.0.1:8080/nifi-api/processors/{id} or http://127.0.0.1:8080/nifi-api/flow/bulletin-board ) checking the concrete component or check bulletin-board for messages on this component. Maybe my idea is absurd - I am sure hundreds of you solved this problem already. Any kind of help is very welcome! Using NiFi 1.6.0 ... View more

Hi @Artjom Zab I'm facing the same issue and followed the same links as you. Because I'm quite new to NiFi my opinion might not be relevant but here are my thoughts on errorhandling. Similar to your description I have built a processgroup where I handle the logic of retry for 3 times. Because the output port has to be tied to the original processor I don't see any other solution then connecting this processgroup to every retrieable action/processor. Because on my installation is the NiFi-Registry available I built the processgroup one time and import it on every place from the registry. But if there will be changes one day on this processgroup there is a lot of work to do exchanging the version of this very frequently used processgroup. BTW: In my first concrete case there is no worry about data loss relating to flowfiles which failed (with or without retry) because after informing the administrator about the details of the problematic data and his solution of the problem there always have to be requested actual data from the source systems for rerun. Would be glad getting some tips for a better solution. Bye! ... View more

@Shu Thanks for Your answer! Using NiFi 1.6.0 --- must have been blind - indeed I haven't seen this search option, sorry. Bye! ... View more

@Matt Burgess Superb, it works perfect! One day I will try your original solution, sounds interesting. Bye! ... View more

@Matt Burgess Great, my boss and I are very grateful for your "very specific" help! One last question concerning the script: The perfect solution would leave the FF-content unchanged and deliver the maxOutput as FF-Attribute. Tried this but can't find the right syntax: flowFile = session.putAttribute(flowFile, 'MAX_geaendertAm_ABAS', ${max.geandertAm_ABAS}) Is this possible? ... View more

@Matt Burgess Thanks for your quick response! Sorry, but I'm afraid I'm unable to cope with your answer. Knowing NiFi only for a few weeks, knowing nothing about the configuration of the Controller Services for JsonPathReader/JsonRecordSetWriter and need to solve the described problem. I was hoping on some script solution like... flowFile = session.get() if(!flowFile) return flowFile = session.putAttribute(flowFile, 'Value_Groovy', (FF_content.geaendertAm_ABAS.max())) session.transfer(flowFile, REL_SUCCESS) ... where FF_content is an attribute which contains the json. No solution for "beginners" possible? ... View more