Member since
12-03-2017
147
Posts
24
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1156 | 11-03-2023 12:17 AM | |
2891 | 12-12-2022 09:16 PM | |
1096 | 07-14-2022 03:25 AM | |
1750 | 07-28-2021 04:42 AM | |
2020 | 06-23-2020 10:08 PM |
10-03-2024
04:32 AM
1 Kudo
Hello Experts, We have 2 node nifi cluster running on k8 cluster. We want to distribute the incoming http request on specific port to be load balanced across both nodes equally (round robin), do we have anyway in kubernetes for this? Tried headless service & ClusterIP but did not work as expected. Is there any other way to achieve this without external load balancers like AWS ELB etc. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
09-09-2024
01:52 PM
@hegdemahendra The FlowFile connection back pressure thresholds are soft limits. Once one of the configured back pressure thresholds on reached or exceeded, NiFi will not allow the processor feeding that connection to get scheduled to execute again. So in your case no back pressure is being applied, so the ConsumeAzureEventHub processor is being allowed to scheduled to execute. During a single execution it is consuming more events then the threshold settings. What is the batch size set to in your ConsumeAzureEventHub processor? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-04-2024
10:18 PM
1 Kudo
@araujo @bbende @MattWho - do you have any suggestions?
... View more
09-04-2024
07:54 AM
Hello @Mais - Were you able to deserialise and consume both key & value ? In my case I am able to get deserialised value but dont see key anywehere!
... View more
05-30-2024
11:24 PM
1 Kudo
What an explanation ! Cleared my doubts. Thank you so much @MattWho .
... View more
05-20-2024
02:01 PM
1 Kudo
Hello @hegdemahendra Always very helpful if you include the exact version of Apache NiFI, Cloudera HDF, or Cloudera CFM being used. My guess here would be one or both of the following: You have multiple FlowFiles all pointing at the same content claims queued in connections within your dataflow(s) on the canvas. As long as a FlowFile exists on the canvas it will exist in flowfile_repository. Users should avoid leaving FlowFiles queued in connection on NiFi. Some users tend to allow FlowFile to accumulate at stopped processor components rather then auto-terminate them. Even if a FlowFile does not have any content its FlowFile attributes/metadata still consume disk space. You are extracting content from your FlowFiles into FlowFile attributes resulting in large FlowFile attribute/metadata being stored in the flowfile_repository. Dataflow designers should avoid extracting large amounts flowfile content in to the FlowFile's attributes. Instead try to build dataflows and utilize components that read content from the FlowFile's content instead of from FlowFile attributes. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-23-2024
01:46 AM
1 Kudo
Not possible, there is a reason for that if we allow to add a property to set an initial offset, every time that processor was stopped and started we would again start at that configured offset, with console consumer, you only pass the initial offset on the first execution then subsequent executions use the last stored offset in Kafka, which is not the case here with the processor. Thus only two options are "earliest" or "latest". Thank you
... View more
03-28-2024
09:51 AM
1 Kudo
@hegdemahendra You may try Cloudera Hive JDBC Driver. The driver class name would be "com.cloudera.hive.jdbc.HS2Driver".
... View more
03-26-2024
07:33 AM
hi @hegde , thank you for your fast reply. This would have worked indeed. Fortunately, we found the culprit on our matter. Someone made manual changes to the nifi github repo which was used by the nifi-registry. We reverted the changes in github and now the registry seems to work again 🙂
... View more
03-12-2024
12:20 AM
Hello Experts, I have a custom processor where I have below lines of code - private ReadContext configJsonCtx = JsonPath.parse(configJsonString); private List<Object> workFlows = configJsonCtx.read("$.Workflow"); this.workFlows.forEach((workFlowObj) -> { try { DocumentContext workFlow = JsonPath.parse(workFlowObj); ........ This is working fine with Nifi 16 and com.jayway.jsonpath.json-path:2.4.0 dependency. But now I am trying to upgrade to nifi 24 and I am getting below error when I do that Any idea what could be the reason and solution for the same. 2024-03-12 05:09:08,815 ERROR [Timer-Driven Process Thread-320] c.o.n.p.o.O9QueueRouting O9QueueRouting[id=8718e632-f033-372b-8f6c-682399fbf9b6] Exception: java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.lambda$onScheduled$0(O9QueueRouting.java:299) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.onScheduled(O9QueueRouting.java:269) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55) at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1765) at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Thanks in advnace Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi