Thank You @Matt Burgess and @greyKeys . Previously I used timestamp which results in error, then I changed it to ${uuid} its working fine. ... View more

big thanks really... ... View more

Thanks @Matt Burgess. Currently I am handling this using a Javascript, similar approach to what you described. I wanted to confirm there is no other way. For simpler structures, I managed to extract the key values using Regex, but for deep nested keys, I was forced to use ExecuteScript. ... View more

@Matt Burgess @Arsalan Siddiqi Now Logs are changed 2016-11-17 10:01:04,640 ERROR [StandardProcessScheduler Thread-3] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1392) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode.access$100(StandardProcessorNode.java:90) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1230) ~[na:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_91] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_91] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_91] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91] Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_91] at java.util.concurrent.FutureTask.get(FutureTask.java:206) [na:1.8.0_91] at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1375) ~[na:na] ... 9 common frames omitted Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor301.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1234) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1230) ~[na:na] ... 6 common frames omitted Caused by: java.lang.IllegalStateException: Cannot invoke method public abstract java.sql.Connection org.apache.nifi.dbcp.DBCPService.getConnection() throws org.apache.nifi.processor.exception.ProcessException on Controller Service with identifier 47e0b89a-2e72-46d5-956e-f3b2f2b32a3f because the Controller Service is disabled at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:161) ~[na:na] at com.sun.proxy.$Proxy91.getConnection(Unknown Source) ~[na:na] at org.apache.nifi.processors.standard.QueryDatabaseTable.setup(QueryDatabaseTable.java:225) ~[na:na] ... 14 common frames omitted ... View more

Thanks @Andrew Grande! That worked! I feel like a noob 🙂 but appreciate all the help! ... View more

now use record processor ... View more

@mclark @Matt Burgess so in any case does the file needs to be read in to memory before it splits.?? either by lines or by bytes. i was hoping it starts the next process work once it receive first split.? in my case it waited 8 minutes until it split the 10GB file into 1200+ splits. If my files are about 100 GB each (I have 18 such files) I am scared to run the whole flow for all files. I may have to run for each file one by one.? ... View more

I am getting same error in HDP 3. ... View more

Great answer! Just to add a caveat, if you are using HDF 2.0 and HDP 2.5, please see the following: https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html ... View more