Apache Flow file from PUTSQL processor is not sending to Failed or Retry queue

[#1]

Hi , I have large data and using PUTSQL processor inserting into the database. While inserting the data, i have observed some failures saying "IndexOutOfBoundException". The below is the exception.

2017-04-08 13:53:01,181 INFO [Provenance Maintenance Thread-3] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 9687527 2017-04-08 13:53:04,197 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.standard.PutSQL PutSQL[id=1272118f-17d0-1471-23dd-2e94d6a1f1f6] PutSQL[id=1272118f-17d0-1471-23dd-2e94d6a1f1f6] failed to process due to java.lang.IndexOutOfBoundsException: Index: 1, Size: 1; rolling back session: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 2017-04-08 13:53:04,197 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.standard.PutSQL java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_111] at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_111] at org.apache.nifi.processors.standard.PutSQL.onTrigger(PutSQL.java:304) ~[nifi-standard-processors-1.1.0.jar:1.1.0] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.0.jar:1.1.0] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0.jar:1.1.0] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.0.jar:1.1.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] 2017-04-08 13:53:04,207 ERROR [Timer-Driven Process Thread-6] o.apache.nifi.processors.standard.PutSQL PutSQL[id=1272118f-17d0-1471-23dd-2e94d6a1f1f6] PutSQL[id=1272118f-17d0-1471-23dd-2e94d6a1f1f6] failed to process session due to java.lang.IndexOutOfBoundsException: Index: 1, Size: 1: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1

Connected FAILED and RETRY queues from PUTSQL to PUTFILE to capture the query when its failed. But when PUTSQL failed, flowfile is not sendingto the queue by PUTSQL. Please let us know how can i see the failed query in this scenario??

One more question:-

When i did Data Provenance on the PUTSQL processor, its not showing the current data and seems to be showing only 1000 entried. Is there any way to see the full data for a particular processor??

Thanks

Ravi

[#2]

@Ravi Teja

The flowfile isn't going out the FAILED connection because the session is rolled back to the incoming queue. I have highlighted that part of the log with the roll back information. Meaning the flowfile is penalized and then put back on the incoming queue for the PutSQL processor.

If you want to write out the file that is causing the roll back, with a PutFile processor, there are ways to pull that flowfile out of the flow. You can use an RouteOnAttribute and put the uuid of the flowfile in as a property to get the flowfile and then route the flowfile out to the PutFile processor.

For provenance events, there is a slight delay in the writing of events based on the configuration of NiFi. This is the property that controls how quickly the provenance events are available in the UI:

nifi.provenance.repository.rollover.time -- The amount of time to wait before rolling over the latest data provenance information so that it is available in the User Interface. The default value is 30 secs. To see events for a particular processor, right click on the processor and a menu will pop up, select Data provenance

Then another window will open displaying the provenance events for just that processor