About hegdemahendra

hegdemahendra · ‎03-08-2024

Thank you so much @MattWho !

Babasaheb · ‎03-06-2024

Hello @hegdemahendra, 1) Please refer to the following article to connect Kafka from Nifi: https://community.cloudera.com/t5/Community-Articles/Integrating-Apache-NiFi-and-Apache-Kafka/ta-p/247433 2) Also, to isolate the issue you can try to connect Kafka from the same settings from nifi node using the Kafka command Please let us know if you still have any questions regarding the same or facing any issues. We will be happy to assist you with it. If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post. Thank you.

hegdemahendra · ‎02-22-2024

Thank you very much for the quick reply @cotopaul @MattWho . 2024-02-22 14:44:24,319 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 4290 2024-02-22 14:44:25,086 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootstrap requests on port 44846 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Apache NiFi _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut _ __ ___ __ _(_)___| |_ _ __ _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | '__/ _ \/ _` | / __| __| '__| | | | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | | | __/ (_| | \__ \ |_| | | |_| | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut |_| \___|\__, |_|___/\__|_| \__, | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut ==========|___/================|___/= 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut v 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:31,494 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Server start failed: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'standardServiceFacade' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/service/StandardServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseMetadataService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/db/DatabaseMetadataService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.license.FlywayEditionUpgradeRequiredException: Flyway Teams Edition or MySQL upgrade required: MySQL 5.7 is no longer supported by Flyway Community Edition, but still supported by Flyway Teams Edition. 2024-02-22 14:44:32,331 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry ^C root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# mysql --version mysql Ver 14.14 Distrib 5.7.33, for Linux (x86_64) using EditLine wrapper root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# I see above logs in bootstrap, Is the issue because my mysql version is 5.7? In another server I am running nifi 1.16 registry with this mysql 5.7 version and its working there from 2 years. Thanks Mahendra

hegdemahendra · ‎02-05-2024

Thanks for the quick reply @MattWho . We found the issue. It was due to firewall setup for the partition-2 on broker side. When they fixed it started working

MattWho · ‎12-01-2023

@hegdemahendra You also need to be careful with some rest-api request as they may generate reports that are held in NiFi heap until a rest-api call is made to remove them. Doing similar actions directly from the UI handles the multiple calls needed automatically. So calls like these can slowly eat away at the NiFi heap impacting performance until a restart. Provenance queries would be an example of this. Unrelated note: NiFi Variable registry has been deprecated in favor of NiFi Parameters now. The Variable registry functionality has been officially removed starting with the latest NiFi 2.0.0-M1 release. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt

hegdemahendra · ‎11-03-2023

Found the issue, DisrtibtedMapCacheServer was configured with default 1 mb to handle in each cache request. But I was trying to pass a attribute value to pass to wait processor from Notify which was 1.5 mb. After increasing the cache server config from 1mb to 10 mb it worked

hegdemahendra · ‎10-16-2023

Hello experts, I am using Apache nifi to connect to Delta table and read and write data. Nifi connected to Delta table via Spark Thrift server jdbc. (I use one custom processor and ConvertJsonToSql processor) Now the issue is - when I create a table with below create script - create table schemagetnifi.getexternal_nifi1 (userId integer, id integer, title String, completed boolean); it works fine. But if I use below table create statement create table schemagetnifi.getexternal (userId integer, id integer, title String, completed boolean) using delta; Nifi (java) is throwing error (looks like its unable read meta data) Only difference in 2 different tables creation script is "USING DELTA", with that I am facing error. Any suggestion or help would be much appreciated

RangaReddy · ‎10-05-2023

Hi @hegdemahendra CDP onwards Spark Thrift Server is not supported. You can try the following links maybe it is useful for you: https://stackoverflow.com/questions/29227949/how-to-implement-spark-sql-pagination-query

MattWho · ‎10-04-2023

@hegdemahendra The autoload capability in NiFi can only auto-load new nars added to the directory. It does not handle unload or reload. The reason for this is because a reload would require the upgrade of existing components using a previously uploaded nar. This process would require the stopping of all components added to canvas from that nar, upgrading all those components to the new nar version, and then starting the components again. You also have the issue with the fact that the flow.json.gz has already been loaded in to memory with a different component version. Then you also have the issue of when someone adds a new nar version and does not remove the old nar version first. You should be able to click on a component on the canvas once multiple version of same class are loaded and switch to the other version. With the way NiFi is designed, NiFi will allow multiple versions of the same components to be loaded (always been that way). So there has never been the capability when multiple versions of the same component are loaded to trigger an upgrade of any components from those duplicate component classes on the canvas. NiFi can only change a component's version on startup and only if only one version of the component class exists on startup. On startup, NiFi loads the NiFi lib nars and any nars found in custom lib folders or autoload directory. These nars get unpacked in to a work directory. NiFi then starts loading the dataflow from the flow.json.gz file. The flow.json.gz contains each components class, version, and configuration. When loading a component where version is not found but ONLY one different version of that same component class is found, NiFi will switch that component that version of the class (could be older version or newer). If any component versions changed on startup a new flow.json.gz is written out to match what is loaded in to memory. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt

hegdemahendra · ‎07-18-2023

Old version of Databricks jdbc jar has the simba driver class, with that I am able to use jdbc:spark:// connection string. But jdbc:hive2:// performance looks better with thrift for me

Online	Offline
Last Visited	‎11-21-2024 10:16 AM

Member Since	‎12-03-2017 03:34 AM
Last Visited	‎11-21-2024 10:16 AM
Posts	147
Kudos received	24

Cloudera Community

Re: Notify processor throws "channel unregistered ...

Re: java.lang.NoClassDefFoundError: org/apache/nif...

Re: org.apache.nifi.util.TestRunners giving error ...

Re: NiFi registry commits templates to git with de...

Re: Convert Integer type value to Date format "yyy...

Re: Nifi is responding SERVICE_UNAVAILABLE when a...

Re: Unable to consume message from kafka broker in...

Re: Nifi registry 1.24 is not starting

Re: Nifi ConsumeKafka_2_6 (nifi 1.16.3) is reading...

Re: Nifi admin rest api becomes slow day by day if...

Re: Notify processor throws "channel unregistered ...

Delta table connection via JDBS (Spark Thrift Serv...

Re: Spark Thrift server JDBC sql with pagination

Re: Nifi autoload custom nar - how to reload the n...

Re: How to access Spark Thrift server via jdbc fro...