Hey guys! Hope you are having a good day! I have a couple of questions regarding error handling in nifi. I found following links to be very useful to learn more about nifi error handling : https://community.hortonworks.com/questions/77336/nifi-best-practices-for-error-handling.html https://community.hortonworks.com/articles/76598/nifi-error-handling-design-pattern-1.html https://pierrevillard.com/2017/05/11/monitoring-nifi-introduction/ So regarding error handling there seem to be two general ways to go: flows that you want to retry flows that you don't want to retry Flows that you want to retry Count the number of retries and if a certain number of retries is reached send an email to the admin. After that wait by using a disabled processor: Flows that you don't want to retry No need to count here, simply send an email and wait for the admin to fix the problem. In both cases, we do not want to drop data that is routed to e.g. failure. Instead we route the data to a disabled processor, to queue it. After fixing the problem, the admin can enable the processor and the data can be processed again. Here is full example (a flow to retrieve an authentication token) What are your thoughts on that? Do you think this kind of error handling is correct/ or can be improved? What bothers me is, that the ERROR process group will be all over the place. However I do not see any way to centralize it (since we have to deal with each failure flow individually, in order not to loose data). ... View more

I want to check if a product ID exists in a file. I came up with following python script: #!/usr/bin/python import os import java.io flowFile = session.get() exists = False if (flowFile != None): if !os.path.exists('/home/nifi/products.csv'): session.transfer(flowFile, REL_SUCCESS) else: asin = flowFile.getAttribute('asin') with open('/home/nifi/products.csv') as csv: if asin in csv.read(): exists = True session.transfer(flowFile, REL_FAILURE) if !exists: session.transfer(flowFile, REL_SUCCESS) However I am getting an error which I find really hard to interpret. It basically just says javax.script.ScriptException: NameError: name 's' is not defined in <script> at line number 9 Any ideas whats causing this? 2018-11-05 19:27:29,267 ERROR [Timer-Driven Process Thread-2] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=e52f46d5-0166-1000-fc42-8e9565527c21] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: NameError: name 's' is not defined in <script> at line number 9: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: NameError: name 's' is not defined in <script> at line number 9 org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: NameError: name 's' is not defined in <script> at line number 9 at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:239) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.script.ScriptException: NameError: name 's' is not defined in <script> at line number 9 at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:222) at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:59) at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31) at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264) at org.apache.nifi.script.impl.JythonScriptEngineConfigurator.eval(JythonScriptEngineConfigurator.java:59) at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:229) ... 10 common frames omitted Caused by: org.python.core.PyException: null at org.python.core.Py.NameError(Py.java:290) at org.python.core.PyFrame.getname(PyFrame.java:257) at org.python.pycode._pyx189507.f$0(<script>:18) at org.python.pycode._pyx189507.call_function(<script>) at org.python.core.PyTableCode.call(PyTableCode.java:171) at org.python.core.PyCode.call(PyCode.java:18) at org.python.core.Py.runCode(Py.java:1614) at org.python.core.__builtin__.eval(__builtin__.java:497) at org.python.core.__builtin__.eval(__builtin__.java:501) at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259) at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:57) ... 14 common frames omitted ... View more