Member since
12-03-2017
149
Posts
25
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1287 | 11-03-2023 12:17 AM | |
3006 | 12-12-2022 09:16 PM | |
1146 | 07-14-2022 03:25 AM | |
1814 | 07-28-2021 04:42 AM | |
2115 | 06-23-2020 10:08 PM |
05-11-2024
02:13 AM
1 Kudo
Hello experts, I am facing an issue in one of the Nifi server where we have multiple consume eventhub flows. The flow file repository disc is getting full but content and provenance repos are not. Have attached the screen shot of all repos usage and content of flowfile repo. journals folder is occupying very large amount of data. nifi.properties (related to flofile repo): nifi.flowfile.repository.always.sync=false nifi.flowfile.repository.checkpoint.interval=2 mins nifi.flowfile.repository.directory=/flowfile nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository nifi.flowfile.repository.partitions=256 nifi.flowfile.repository.retain.orphaned.flowfiles=true nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog Can anyone help me understand what is the issue? how to resolve this? Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
04-22-2024
10:05 AM
Hello Experts, I am using Apache Nifi 1.25 now and have a requirement to consume data from a specific offset of a topic partition. I see in Nifi ConsumeKafka processor to configure only "earliest" or "latest" but not specific offset number, is there any way to achieve this ? Any info/suggestion would be appreciated. Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
03-26-2024
04:29 AM
1 Kudo
As you have already integrated with Git using GitFlowPersistenceProvider, you should have all your latest flows in git. So you can create a new directory on your nifi registry machine and clone the git branch inside the new directory and point nifi reg to that new directory. (Take a backup of metadata db (H2/MySQL/Postgres) before doing this)
... View more
03-26-2024
04:23 AM
1 Kudo
Hello Exeprts,
I am using hive-jdbc jar in my application to connect Hive and Delta (via Thrift server) from Nifi flow using a custom processor -
https://mvnrepository.com/artifact/org.apache.hive/hive-jdbc
Now the problem is this has not released any major version in recent times and all the available older versions have velnurables in it. (Even I have tried '4.0.0-beta-1' , I don't see any velnurables with this, but looks like they have removed some internal dependencies, so it did not work for me)
So to get rid of velnurbles I am checking to see if there is any alternate option for me to use.
Thanks
Mahendra
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache NiFi
03-12-2024
12:20 AM
Hello Experts, I have a custom processor where I have below lines of code - private ReadContext configJsonCtx = JsonPath.parse(configJsonString); private List<Object> workFlows = configJsonCtx.read("$.Workflow"); this.workFlows.forEach((workFlowObj) -> { try { DocumentContext workFlow = JsonPath.parse(workFlowObj); ........ This is working fine with Nifi 16 and com.jayway.jsonpath.json-path:2.4.0 dependency. But now I am trying to upgrade to nifi 24 and I am getting below error when I do that Any idea what could be the reason and solution for the same. 2024-03-12 05:09:08,815 ERROR [Timer-Driven Process Thread-320] c.o.n.p.o.O9QueueRouting O9QueueRouting[id=8718e632-f033-372b-8f6c-682399fbf9b6] Exception: java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.lambda$onScheduled$0(O9QueueRouting.java:299) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.onScheduled(O9QueueRouting.java:269) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55) at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1765) at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Thanks in advnace Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-08-2024
02:20 AM
1 Kudo
Thank you so much @MattWho !
... View more
03-05-2024
08:46 AM
@MattWho - requesting your opinion here (if possible)
... View more
03-04-2024
10:35 PM
1 Kudo
Hello experts, We are using Nifi for data ingestion via http - using 'HandleHttpRequest' processor. Data is getting ingested continuously but once in a while when incoming volume is little high we get 'Service Unavailable - 503' error from nifi end point. Below is the log we see in nifi app logs - 2024-03-04 11:20:25,558 WARN [StandardHttpContextMap-c2ffb1fd-2c09-3a90-0e5e-18384499605f] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=c2ffb1fd-2c09-3a90-0e5e-18384499605f] Request from 10.13.2.204 timed out; responding with SERVICE_UNAVAILABLE Imp configs : [HandleHttpRequest processor] Container Queue Size : 2000 [StandardHttpContextMap controller service] Maximum Outstanding Requests : 8000 Request Expiration : 3min nifi.web.jetty.threads=200 (in nifi.properties) Maximum Timer Driven Thread Count = 200 Total number of running processors on this env : 4200+ This nifi is running on 8 core machine (ec2 instance) When we look at the CPU usage during the issue we dont see a cpu usage is more than 40% Can anyone help me to understand the issue and the fix? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
02-22-2024
06:55 AM
Thank you very much for the quick reply @cotopaul @MattWho . 2024-02-22 14:44:24,319 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 4290 2024-02-22 14:44:25,086 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootstrap requests on port 44846 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Apache NiFi _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut _ __ ___ __ _(_)___| |_ _ __ _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | '__/ _ \/ _` | / __| __| '__| | | | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | | | __/ (_| | \__ \ |_| | | |_| | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut |_| \___|\__, |_|___/\__|_| \__, | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut ==========|___/================|___/= 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut v 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:31,494 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Server start failed: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'standardServiceFacade' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/service/StandardServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseMetadataService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/db/DatabaseMetadataService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.license.FlywayEditionUpgradeRequiredException: Flyway Teams Edition or MySQL upgrade required: MySQL 5.7 is no longer supported by Flyway Community Edition, but still supported by Flyway Teams Edition. 2024-02-22 14:44:32,331 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry ^C root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# mysql --version mysql Ver 14.14 Distrib 5.7.33, for Linux (x86_64) using EditLine wrapper root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# I see above logs in bootstrap, Is the issue because my mysql version is 5.7? In another server I am running nifi 1.16 registry with this mysql 5.7 version and its working there from 2 years. Thanks Mahendra
... View more
02-21-2024
09:23 AM
Hello experts, I have downloaded and trying to run nifi registry 1.24 with meta data db as mysql and Git flow persistence. Getting below error and unable to start the service - log in text : 2024-02-21 16:11:13,716 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry Runtime Java version: 11.0.14 2024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Starting Apache NiFi Registry... 2024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Working Directory: /usr/independent_nifi/nifi-registry-1.24.0 2024-02-21 16:11:13,726 INFO [main] o.apache.nifi.registry.bootstrap.Command Command: java -classpath /usr/independent_ni............................. 2024-02-21 16:11:13,743 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 19099 2024-02-21 16:11:14,447 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootstrap requests on port 37730 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Apache NiFi _ _ 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut _ __ ___ __ _(_)___| |_ _ __ _ _ 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | '__/ _ \/ _` | / __| __| '__| | | | 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | | | __/ (_| | \__ \ |_| | | |_| | 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut |_| \___|\__, |_|___/\__|_| \__, | 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut ==========|___/================|___/= 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut v 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-21 16:11:20,550 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Wed Feb 21 16:11:20 UTC 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2024-02-21 16:11:21,763 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Server start failed: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'standardServiceFacade' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/service/StandardServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseMetadataService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/db/DatabaseMetadataService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.flywaydb.core.Flyway]: Factory method 'flyway' threw exception; nested exception is org.flywaydb.core.api.FlywayException: Unable to obtain connection from Flyway DataSource 2024-02-21 16:11:22,745 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry nifi-registry.properties: nifi.registry.web.war.directory=./lib nifi.registry.web.http.host=nifi12-dev.xyz.com nifi.registry.web.http.port=61080 nifi.registry.web.https.host= nifi.registry.web.https.port= nifi.registry.web.https.application.protocols=http/1.1 nifi.registry.web.jetty.working.directory=./work/jetty nifi.registry.web.jetty.threads=200 nifi.registry.web.should.send.server.version=true nifi.registry.security.keystore= nifi.registry.security.keystoreType=jks nifi.registry.security.keystorePasswd= nifi.registry.security.keyPasswd= nifi.registry.security.truststore= nifi.registry.security.truststoreType=jks nifi.registry.security.truststorePasswd= nifi.registry.security.needClientAuth=True nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml nifi.registry.security.authorizer=managed-authorizer nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml nifi.registry.security.identity.provider= nifi.registry.sensitive.props.additional.keys= nifi.registry.providers.configuration.file=./conf/providers.xml nifi.registry.registry.alias.configuration.file=./conf/registry-aliases.xml nifi.registry.extensions.working.directory=./work/extensions nifi.registry.db.directory= nifi.registry.db.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE;useSSL=false # database properties nifi.registry.db.url=jdbc:mysql://localhost:3306/nifi24_registry nifi.registry.db.driver.class=com.mysql.jdbc.Driver nifi.registry.db.driver.directory=/usr/share/java/ nifi.registry.db.username=registry nifi.registry.db.password=PasswordSample1234 nifi.registry.db.maxConnections=5 nifi.registry.db.sql.debug=false Any suggestion / solution is much appreciated. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry