disconnect and reconnect again this functionality i want to achieve with Publish kafka processor . ConsumerKafka will consumer continuously . ... View more

I know this sounds like batching the problem but after the UpdateRecord can you do JoltTransformRecord to transform "" to null for the target field? here is an example of spec that can do such thing: https://github.com/bazaarvoice/jolt/issues/667   ... View more

@SowmyaP : I encountered some similar issues and it was always related to the HEAP Memory and the CPU consumption. To solve the problem, I stopped the entire NiFi Cluster and started it all over again, making sure that the processors are all in STOPPED state (see Nifi.conf file from /conf for the property). Once you start your cluster back, make sure that you delete the file from the queue, to avoid future problems.  ... View more

Hi @cotopaul , Thanks for the input, can you help on how to setup the invokeHTTP processor and how to fetch all the queues that we were not able to find in the Mongo DB. ... View more

@Wadok88, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. ... View more

Thanks a lot! @cotopaul  Solution 1 worked in our case. We stopped our processors and emptied queues, after that moving the connection from center funnel to the new processor helped us to remove other funnels. To answer your question, why the infinite loop ? it was effect of blunder - when I was working with the funnels weren't located so sparsely, so I didn't see that the connection I'm going to make and created a loop ... View more

@user001, I do not know what your Sandbox environment is, but I just tested with NiFi 1.15 and the solution works like a charm. So either you are doing something wrong, or you have something configured incorrectly in your sandbox environment. Or maybe NiFI 1.18 is bugged - but I highly doubt it, as there would have been far more posts reporting similar issues. ... View more

Thanks for the answer. following is the complete stack trace log. 2023-10-10 14:01:01,542 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@15e0fe05{nifi-api,/nifi-api,file:///opt/exgw/nifi-1.17.0/work/jetty/nifi-web-api-1.17.0.war/webapp/,AVAILABLE}{./work/nar/extensions/nifi-server-nar-1.17.0.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.17.0.war} 2023-10-10 14:01:01,703 ERROR [Timer-Driven Process Thread-1] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=22ec0a77-b91f-3a2a-e19b-dab6e46e0717] Request Processing failed: StandardFlowFileRecord[uuid=893fd559-fd28-475b-b2d1-e94fbf566d86,claim=,offset=0,name=893fd559-fd28-475b-b2d1-e94fbf566d86,size=0] org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from buffer(ResponseBodySource(okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource@1de64fd6)).inputStream() for StandardFlowFileRecord[uuid=f4327acc-d1ea-4450-a5ad-89723bfafec6,claim=,offset=0,name=893fd559-fd28-475b-b2d1-e94fbf566d86,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.EOFException at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3580) at org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:1008) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1356) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.EOFException at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3573) ... 13 common frames omitted Caused by: java.io.EOFException: null at okio.RealBufferedSource.require(RealBufferedSource.kt:199) at okio.RealBufferedSource.readHexadecimalUnsignedLong(RealBufferedSource.kt:381) at okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.kt:429) at okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.kt:408) at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:276) at okio.RealBufferedSource$inputStream$1.read(RealBufferedSource.kt:158) at java.io.InputStream.read(InputStream.java:101) at org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62) at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:837) at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3570) ... 13 common frames omitted ... View more

@MWM @cotopaul   If you get the record reader/writer using the schema(s) you want,  you do not have to do any magic to convert values, it should just work.     Only use, inferSchema long enough to get the structure when you have none.  Then copy/paste it and use it as @cotopaul has described in place of InferSchema.  You can also use Schema Registry.  Make the edits you need to satisfy reader (upstream), writer (downstream) as they are sometimes needing minor adjustments like in this case. ... View more