Cloudera Community
hegdemahendra
user rank
Master Collaborator

Re: Nifi autoload custom nar - how to reload the n...

by Master Mentor Master Mentor in Support Questions
‎10-04-2023 05:46 AM
‎10-04-2023 05:46 AM
@hegdemahendra  The autoload capability in NiFi can only auto-load new nars added to the directory.  It does not handle unload or reload.   The reason for this is because a reload would require the upgrade of existing components using a previously uploaded nar.  This process would require the stopping of all components added to canvas from that nar, upgrading all those components to the new nar version, and then starting the components again.  You also have the issue with the fact that the flow.json.gz has already been loaded in to memory with a different component version. Then you also have the issue of when someone adds a new nar version and does not remove the old nar version first.  You should be able to click on a component on the canvas once multiple version of same class are loaded and switch to the other version.    With the way NiFi is designed, NiFi will allow multiple versions of the same components to be loaded (always been that way).  So there has never been the capability when multiple versions of the same component are loaded to trigger an upgrade of any components from those duplicate component classes on the canvas. NiFi can only change a component's version on startup and only if only one version of the component class exists on startup.  On startup, NiFi loads the NiFi lib nars and any nars found in custom lib folders or autoload directory.  These nars get unpacked in to a work directory.  NiFi then starts loading the dataflow from the flow.json.gz file.  The flow.json.gz contains each components class, version, and configuration.  When loading a component where version is not found but ONLY one different version of that same component class is found, NiFi will switch that component that version of the class (could be older version or newer).  If any component versions changed on startup a new flow.json.gz is written out to match what is loaded in to memory. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Re: How to access Spark Thrift server via jdbc fro...

by Master Collaborator in Support Questions
‎07-18-2023 08:46 PM
‎07-18-2023 08:46 PM
Old version of Databricks jdbc jar has the simba driver class, with that I am able to use jdbc:spark:// connection  string. But jdbc:hive2:// performance looks better with thrift for me ... View more

Re: Concurrent Tasks cant be changed in ConsumeAz...

by Master Collaborator in Support Questions
‎07-16-2023 09:37 PM
‎07-16-2023 09:37 PM
@MattWho  Its very clear now, thank you so much ! ... View more

Re: How to integrate Nifi with delta table (delta ...

by Master Mentor Master Mentor in Support Questions
‎07-10-2023 01:01 PM
‎07-10-2023 01:01 PM
@hegdemahendra  There currently are no NiFi processors for ingesting from Delta Lake.  There is an existing Apache Jira for creating such list/fetch processors for Delta Lake, but it is still open and does not look to be actively being worked on: https://issues.apache.org/jira/browse/NIFI-7093 I am not aware of any open source 3rd party options here as well. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

Re: Nifi - confluent cloud kafka broker connectivi...

by Master Collaborator in Support Questions
‎07-07-2023 09:18 AM
‎07-07-2023 09:18 AM
@MattWho Thank you so much for this awesome detailed solution ! It worked. ... View more

Re: Apache nifi-api is taking too long time to upd...

by Master Mentor Master Mentor in Support Questions
‎02-02-2023 01:04 PM
‎02-02-2023 01:04 PM
@hegdemahendra  When updating a NiFi variable a serious of steps needs to occur. Steps To Update Variables Identifying components affected Stopping affected Processors Disabling affected Controller Services Applying Updates Re-Enabling affected Controller Services Restarting affected Processors So in this process what can lead to this taking a long time to complete is the Stopping of processors and disabling of controller services using that updated variable.  When NiFi requests a component to stop it transitions to a "stopping" or "disabling" stage.  During this phase the component will not linger be scheduled and the process waits for any existing threads being executed by those components to complete. Those threads do not get interrupted.  So when this take aa long time or "infinite" amount of time, troubleshooting this would require getting a series of thread dumps to see which threads are long running or perhaps hung preventing the impacted components from competing the thread execution that blocks the component from transition to a fully stopped or disabled state.  Understand that nothing in  a thread dump is going to directly point back to a very specific processor.  So it is important in your troubleshooting that you know what processors use the variable you are updating and look for threads that appear in the complete series of multiple thread dumps that relate back to those component classes.  Also in cases were you see these long running calls, are they for a variables used consistently by the same set of components to help narrow your analysis. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt    ... View more

Re: Will Nifi slows down with increase in number o...

by Master Mentor Master Mentor in Support Questions
‎02-02-2023 12:06 PM
1 Kudo
‎02-02-2023 12:06 PM
1 Kudo
@hegdemahendra  All the dataflow(s) you construct in the NiFi UI and all the controller services reporting tasks, and templates you create all reside within your NiFi's heap memory.  So the more you build the more heap that is being consumed there.  You also end up with more components being scheduled.  A scheduled processor will need a thread to check it's inbound connection or connect to an external service to check for new FlowFile or data to consume/execute upon.  All these components then need to share the available configured threads from NiFi thread pool.  A running processor is not idle.  It still has a configured run schedule to which it uses to check for work (granted that with n.  work to do those threads are extremely short lived). The default size of the Timer Driven Thread pool is only 10. So as your flow gets larger, I'd recommend looking at Garbage Collection (GC) stats for your NiFi's JVM to see how often GC is happening and how long those GC events take (All GC is stop the world, so no processing happens during GC). I'd also examine your CPU load average and is it is low, increase the size of the Max Timer Driven Thread pool (Global menu --> controller settings --> general) in small increments to see what impact that has on yoru performance.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

Nifi is unable to connect Hive table - FAILED: Exe...

by Master Collaborator in Support Questions
‎12-22-2022 05:52 AM
‎12-22-2022 05:52 AM
Hi Experts,   2022-12-22 10:47:41,012 ERROR [Timer-Driven Process Thread-131] c.o.n.p.o.TrackRequestStatus TrackRequestStatus[id=687e8099-cd28-3c8b-f788-cf64094bd870] ~(5196, 5196, 20582) ~ 5ed6349b-7eb8-49ab-ad42-963c82fa8bc9 ~ 5196.11730922 ~ Error updating track request status - Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 3, vertexId=vertex_1671455880373_0219_136_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:2, Vertex vertex_1671455880373_0219_136_02 [Reducer 3] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1671455880373_0219_136_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:2, Vertex vertex_1671455880373_0219_136_01 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1671455880373_0219_136_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1671455880373_0219_136_00 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:3 java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Dag received [DAG_TERMINATE, SERVICE_PLUGIN_ERROR] in RUNNING state.Error reported by TaskScheduler [[2:LLAP]][SERVICE_UNAVAILABLE] No LLAP Daemons are runningVertex killed, vertexName=Reducer 3, vertexId=vertex_1671455880373_0219_136_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:2, Vertex vertex_1671455880373_0219_136_02 [Reducer 3] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1671455880373_0219_136_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:2, Vertex vertex_1671455880373_0219_136_01 [Reducer 2] killed/failed due to:DAG_TERMINATED]Vertex killed, vertexName=Map 1, vertexId=vertex_1671455880373_0219_136_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_TERMINATED, failedTasks:0 killedTasks:1, Vertex vertex_1671455880373_0219_136_00 [Map 1] killed/failed due to:DAG_TERMINATED]DAG did not succeed due to SERVICE_PLUGIN_ERROR. failedVertices:0 killedVertices:3 at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401) at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266) at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:101) at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94) at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240) at com.sun.proxy.$Proxy412.execute(Unknown Source) at com.o9solutions.nifi.services.TrackRequestStatusHiveDatabaseServiceImpl.saveStatus(TrackRequestStatusHiveDatabaseServiceImpl.java:114) at com.o9solutions.nifi.processors.o9_custom_processors.TrackRequestStatus.onTrigger(TrackRequestStatus.java:286) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:103) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)   Thanks, Mahendra ... View more
Labels:

Re: java.lang.NoClassDefFoundError: org/apache/nif...

by Master Collaborator in Support Questions
‎12-12-2022 09:16 PM
1 Kudo
‎12-12-2022 09:16 PM
1 Kudo
@steven-matison - thank you.   Found out the reason for NoClassdef error it was due to the missing nar dependency in my NAR generation project pom. Need to add "nifi-mongodb-nar" dependency to make use of these mongo related classes as they are not part of standard nar. But I am already using "nifi-standard-nar" as dependency in NAR project I cant use one more, so had drop down this idea and write my own Mongo controller service.   ... View more

Re: Nifi how to sql join two flowfiles

by New Contributor in Support Questions
‎11-29-2022 11:25 PM
‎11-29-2022 11:25 PM
Hi @SirV  can you please send me the code , m also stuck in the same situation.   ... View more
Contact Me
Online
Offline
Last Visited
‎12-20-2024 12:26 PM
Community Statistics
Member Since ‎12-03-2017 03:34 AM
Last Visited ‎12-20-2024 12:26 PM
Posts 149
Kudos received 26