Member since
12-03-2017
156
Posts
26
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2078 | 11-03-2023 12:17 AM | |
4147 | 12-12-2022 09:16 PM | |
1562 | 07-14-2022 03:25 AM | |
2389 | 07-28-2021 04:42 AM | |
3327 | 06-23-2020 10:08 PM |
07-29-2022
07:29 PM
Thank you @MattWho . - 0 flow file in outbound connection. - no backpressure - nifi 1.12.1 version we are using - Don't see any logs related to this - We are using 500 gb disk and only 20gb was utilized at that time But all repos (flowfile, content and prov) are located in same ebs volume of 500gb, so is this could be something due to lack of iops availability?
... View more
07-28-2022
12:28 AM
Hello Experts, We have HandleHttpRequests processor running with 16 concurrency. Container queue size configured is 1000. While running a load test suddenly HandleHttpRequests stopped emitting flowfiles recieved and started throwing error show in below image - So we stopped the load test and now even after 30 mins when we send requests it will just emit 2 or 3 flow files and again shows same error and does not release/emit any request flowfiles. Ideally when no request is getting hit to that port (which HttpRequestListener is configured) container should get cleared and flow files should get created from queue and emit right? What could be the reason this processor is just trhowing container full error and not emitting any flow files ? It started working only when we manually cleanup all repositories (content, provenance, ff) and restart the nifi ! Any suggestion would be appreciated. Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
07-25-2022
09:40 PM
Experts, I am continuously seeing this log in Nifi app log - 2022-07-25 06:12:40,788 INFO [Timer-Driven Process Thread-219] o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup Any idea what is the reason and will there be any impact of this ? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
07-14-2022
03:25 AM
testRunner.setAllowSynchronousSessionCommits(true); Helped me to get rid of this error
... View more
07-13-2022
01:14 AM
Hello Experts, I am using org.apache.nifi.util.TestRunners in Nifi custom processor unit test cases, and code something like below.. testRunner = TestRunners.newTestRunner(O9QueueRouting.class); . . . . MockFlowFile ff = this.testRunner.enqueue(new ByteArrayInputStream(ffContents.getBytes(StandardCharsets.UTF_8))); Map<String, String> attrs = new HashMap<String, String>(); attrs.put("routingheader", "headervalue"); attrs.put("kafka.topic", "topic1"); ff.putAttributes(attrs); testRunner.run(); It was working earlier now started giving error like below - [ERROR] O9QueueRoutingTest.test:73 java.lang.RuntimeException: As of version 1.14.0, ProcessSession.commit() should be avoided when possible. See JavaDocs for explanations. Instead, use commitAsync(), commitAsync(Runnable), or commitAsync(Runnable, Consumer<Throwable>). However, if this is not possible, ProcessSession.commit() may still be used, but this must be explicitly enabled by calling TestRunner. Any idea how this can be resolved ? How to explicitly enable by calling TestRunner ? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
07-09-2022
05:24 AM
It depends on your use case - how many tables, how much records each table has, one time migration vs cdc etc. For one time migration (for till date) , if data is huge then its better to run sql paginated query instead of full table select so that resulting flow file will not be very huge and hence no impact on nifi performance. If you have multiple tables then may be you can add a generate flow file processor for triggering the flow with all the tables names in json and split the json and send each table name to ExecuteSQLQuery processor to execute and get data.. If CDC need migrated then may be you can schedule the flow very frequently and query the data based on datetime range if possible or any incremental values (which can be cached using nifi cache for new run)
... View more
07-09-2022
04:20 AM
Hello Experts, I am upgrading nifi from 1.12.x to 1.16.x and for some time I need to support both the versions. So I am trying to have profiles for nifi 12 and 16 in maven pom so that based on the profile passed it can build the nar file for specific version. I am facing an challenge in above step, because of a refactoring happened in nifi security utils post 1.12. I was using 'OkHttpClientUtil' in 1.12 and which is no longer available in 1.16 So now to support both the version for a custom processor I need to have 2 java classes, one for 1.12 and one for 1.16 with minor code changes. But I want to keep my custom processor name same across both the version, is it possible ? Ex : my java classes like below ... Nifi 12 - ResourceRouting.java Nifi 16 - ResourceRoutingV2.java Is it possible to have custom processor name as "ResourceRouting" ResourceRoutingV2.java also? Any suggestion would be much appreciated. @araujo @MattWho @GangWar @smdas Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
07-05-2022
10:50 PM
1 Kudo
Issue created : https://issues.apache.org/jira/browse/NIFI-10197
... View more
07-01-2022
11:39 AM
Hello Experts, I am trying to upgrade nifi from 1.12.1 version to 1.16.3. I have a script 'ECMAScript' which was working fine in 1.12.1and now behaving different in 1.16.3. Below is the portion of code giving error in 1.16 version when 'xyzId' is null. But it just logs null as expected in 1.12 version var xyzId = flowFile.getAttribute('xyzId'); log.info(xyzId); Below is the error in 1.16 version when null value received for above line (log.info() line throws below error) imer-Driven Process Thread-8] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=bae38131-0181-1000-29af-196312cdd9d9] ExecuteScript[id=bae38131-0181-1000-29af-196312cdd9d9] failed to process due to org.apache.nifi.processor.exception.ProcessException: java.lang.RuntimeException: java.lang.NoSuchMethodException: Can't unambiguously select between fixed arity signatures [(org.apache.nifi.logging.LogMessage), (java.lang.String)] of the method org.apache.nifi.controller.TerminationAwareLogger.info for argument types [null]; rolling back session org.apache.nifi.processor.exception.ProcessException: java.lang.RuntimeException: java.lang.NoSuchMethodException: Can't unambiguously select between fixed arity signatures [(org.apache.nifi.logging.LogMessage), (java.lang.String)] of the method org.apache.nifi.controller.TerminationAwareLogger.info for argument types [null] at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:245) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: Can't unambiguously select between fixed arity signatures [(org.apache.nifi.logging.LogMessage), (java.lang.String)] of the method org.apache.nifi.controller.TerminationAwareLogger.info for argument types [null] at jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397) at jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449) at jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406) at jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402) at jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155) at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264) at org.apache.nifi.script.impl.JavascriptScriptRunner.run(JavascriptScriptRunner.java:39) at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:228) ... 11 common frames omitted Caused by: java.lang.NoSuchMethodException: Can't unambiguously select between fixed arity signatures [(org.apache.nifi.logging.LogMessage), (java.lang.String)] of the method org.apache.nifi.controller.TerminationAwareLogger.info for argument types [null] at jdk.internal.dynalink.beans.OverloadedMethod.throwAmbiguousMethod(OverloadedMethod.java:225) at jdk.nashorn.internal.scripts.Script$299$\^eval\_.:program(<eval>:13) at jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637) at jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494) at jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393) ... 18 common frames omitted Any idea how to fix this? I want to avoid checking null values for logging also Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
02-03-2022
09:13 AM
Hello Experts, I am looking for a regex to configure in "RouteOnContent" processor of Nifi so that based on specific string in flow file content I can route FF. FF content : ● rclone.service - Rclone(file transfer tool) Remote Service
Loaded: loaded (/etc/systemd/system/rclone.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-02-03 09:46:01 UTC; 6h ago
Main PID: 23647 (rclone)
Tasks: 13
Memory: 61.3M
CPU: 5.143s
CGroup: /system.slice/rclone.service
└─23647 /usr/bin/rclone rcd --rc-no-auth --rc-job-expire-duration=60m --cache-tmp-upload-path=/tmp/rclone/upload I want to search for the string "Active: active (running)" in flow file content, if present route to different path. I tried few regex but did not work as expected. Any suggestion would be much appreciated. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi