Member since
12-03-2017
147
Posts
24
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1162 | 11-03-2023 12:17 AM | |
2893 | 12-12-2022 09:16 PM | |
1100 | 07-14-2022 03:25 AM | |
1752 | 07-28-2021 04:42 AM | |
2023 | 06-23-2020 10:08 PM |
03-26-2024
04:29 AM
1 Kudo
As you have already integrated with Git using GitFlowPersistenceProvider, you should have all your latest flows in git. So you can create a new directory on your nifi registry machine and clone the git branch inside the new directory and point nifi reg to that new directory. (Take a backup of metadata db (H2/MySQL/Postgres) before doing this)
... View more
03-26-2024
04:23 AM
1 Kudo
Hello Exeprts,
I am using hive-jdbc jar in my application to connect Hive and Delta (via Thrift server) from Nifi flow using a custom processor -
https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc
Now the problem is this has not released any major version in recent times and all the available older versions have velnurables in it. (Even I have tried '4.0.0-beta-1' , I don't see any velnurables with this, but looks like they have removed some internal dependencies, so it did not work for me)
So to get rid of velnurbles I am checking to see if there is any alternate option for me to use.
Thanks
Mahendra
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
03-12-2024
12:20 AM
Hello Experts, I have a custom processor where I have below lines of code - private ReadContext configJsonCtx = JsonPath.parse(configJsonString); private List<Object> workFlows = configJsonCtx.read("$.Workflow"); this.workFlows.forEach((workFlowObj) -> { try { DocumentContext workFlow = JsonPath.parse(workFlowObj); ........ This is working fine with Nifi 16 and com.jayway.jsonpath.json-path:2.4.0 dependency. But now I am trying to upgrade to nifi 24 and I am getting below error when I do that Any idea what could be the reason and solution for the same. 2024-03-12 05:09:08,815 ERROR [Timer-Driven Process Thread-320] c.o.n.p.o.O9QueueRouting O9QueueRouting[id=8718e632-f033-372b-8f6c-682399fbf9b6] Exception: java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.lambda$onScheduled$0(O9QueueRouting.java:299) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.onScheduled(O9QueueRouting.java:269) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55) at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1765) at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Thanks in advnace Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-08-2024
02:20 AM
1 Kudo
Thank you so much @MattWho !
... View more
03-05-2024
08:46 AM
@MattWho - requesting your opinion here (if possible)
... View more
03-04-2024
10:35 PM
1 Kudo
Hello experts, We are using Nifi for data ingestion via http - using 'HandleHttpRequest' processor. Data is getting ingested continuously but once in a while when incoming volume is little high we get 'Service Unavailable - 503' error from nifi end point. Below is the log we see in nifi app logs - 2024-03-04 11:20:25,558 WARN [StandardHttpContextMap-c2ffb1fd-2c09-3a90-0e5e-18384499605f] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=c2ffb1fd-2c09-3a90-0e5e-18384499605f] Request from 10.13.2.204 timed out; responding with SERVICE_UNAVAILABLE Imp configs : [HandleHttpRequest processor] Container Queue Size : 2000 [StandardHttpContextMap controller service] Maximum Outstanding Requests : 8000 Request Expiration : 3min nifi.web.jetty.threads=200 (in nifi.properties) Maximum Timer Driven Thread Count = 200 Total number of running processors on this env : 4200+ This nifi is running on 8 core machine (ec2 instance) When we look at the CPU usage during the issue we dont see a cpu usage is more than 40% Can anyone help me to understand the issue and the fix? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
02-22-2024
06:55 AM
Thank you very much for the quick reply @cotopaul @MattWho . 2024-02-22 14:44:24,319 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 4290 2024-02-22 14:44:25,086 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootstrap requests on port 44846 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Apache NiFi _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut _ __ ___ __ _(_)___| |_ _ __ _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | '__/ _ \/ _` | / __| __| '__| | | | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | | | __/ (_| | \__ \ |_| | | |_| | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut |_| \___|\__, |_|___/\__|_| \__, | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut ==========|___/================|___/= 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut v 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:31,494 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Server start failed: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'standardServiceFacade' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/service/StandardServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseMetadataService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/db/DatabaseMetadataService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.license.FlywayEditionUpgradeRequiredException: Flyway Teams Edition or MySQL upgrade required: MySQL 5.7 is no longer supported by Flyway Community Edition, but still supported by Flyway Teams Edition. 2024-02-22 14:44:32,331 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry ^C root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# mysql --version mysql Ver 14.14 Distrib 5.7.33, for Linux (x86_64) using EditLine wrapper root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# I see above logs in bootstrap, Is the issue because my mysql version is 5.7? In another server I am running nifi 1.16 registry with this mysql 5.7 version and its working there from 2 years. Thanks Mahendra
... View more
02-21-2024
09:23 AM
Hello experts, I have downloaded and trying to run nifi registry 1.24 with meta data db as mysql and Git flow persistence. Getting below error and unable to start the service - log in text : 2024-02-21 16:11:13,716 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry Runtime Java version: 11.0.14 2024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Starting Apache NiFi Registry... 2024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Working Directory: /usr/independent_nifi/nifi-registry-1.24.0 2024-02-21 16:11:13,726 INFO [main] o.apache.nifi.registry.bootstrap.Command Command: java -classpath /usr/independent_ni............................. 2024-02-21 16:11:13,743 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 19099 2024-02-21 16:11:14,447 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootstrap requests on port 37730 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Apache NiFi _ _ 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut _ __ ___ __ _(_)___| |_ _ __ _ _ 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | '__/ _ \/ _` | / __| __| '__| | | | 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | | | __/ (_| | \__ \ |_| | | |_| | 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut |_| \___|\__, |_|___/\__|_| \__, | 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut ==========|___/================|___/= 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut v 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-21 16:11:20,550 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Wed Feb 21 16:11:20 UTC 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2024-02-21 16:11:21,763 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Server start failed: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'standardServiceFacade' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/service/StandardServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseMetadataService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/db/DatabaseMetadataService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.flywaydb.core.Flyway]: Factory method 'flyway' threw exception; nested exception is org.flywaydb.core.api.FlywayException: Unable to obtain connection from Flyway DataSource 2024-02-21 16:11:22,745 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry nifi-registry.properties: nifi.registry.web.war.directory=./lib nifi.registry.web.http.host=nifi12-dev.xyz.com nifi.registry.web.http.port=61080 nifi.registry.web.https.host= nifi.registry.web.https.port= nifi.registry.web.https.application.protocols=http/1.1 nifi.registry.web.jetty.working.directory=./work/jetty nifi.registry.web.jetty.threads=200 nifi.registry.web.should.send.server.version=true nifi.registry.security.keystore= nifi.registry.security.keystoreType=jks nifi.registry.security.keystorePasswd= nifi.registry.security.keyPasswd= nifi.registry.security.truststore= nifi.registry.security.truststoreType=jks nifi.registry.security.truststorePasswd= nifi.registry.security.needClientAuth=True nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml nifi.registry.security.authorizer=managed-authorizer nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml nifi.registry.security.identity.provider= nifi.registry.sensitive.props.additional.keys= nifi.registry.providers.configuration.file=./conf/providers.xml nifi.registry.registry.alias.configuration.file=./conf/registry-aliases.xml nifi.registry.extensions.working.directory=./work/extensions nifi.registry.db.directory= nifi.registry.db.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE;useSSL=false # database properties nifi.registry.db.url=jdbc:mysql://localhost:3306/nifi24_registry nifi.registry.db.driver.class=com.mysql.jdbc.Driver nifi.registry.db.driver.directory=/usr/share/java/ nifi.registry.db.username=registry nifi.registry.db.password=PasswordSample1234 nifi.registry.db.maxConnections=5 nifi.registry.db.sql.debug=false Any suggestion / solution is much appreciated. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
02-05-2024
07:13 AM
Thanks for the quick reply @MattWho . We found the issue. It was due to firewall setup for the partition-2 on broker side. When they fixed it started working
... View more
02-05-2024
05:13 AM
1 Kudo
Hello experts, We are facing an issue, where preprod Nifi server is able to read a kafka topic correctly. It is able to read all 3 partitions data. At the same time UAT Nifi server is connecting to same topic with different consumer group id and which is reading only the partition 0 & partition 1 but not partition 2. Screen shot from Lenses abt the same consumer group which is facing issue Any idea what could be the possible reasons? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi