Member since
12-03-2017
135
Posts
16
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
334 | 11-03-2023 12:17 AM | |
1811 | 12-12-2022 09:16 PM | |
650 | 07-14-2022 03:25 AM | |
1146 | 07-28-2021 04:42 AM | |
1487 | 06-23-2020 10:08 PM |
03-12-2024
12:20 AM
Hello Experts, I have a custom processor where I have below lines of code - private ReadContext configJsonCtx = JsonPath.parse(configJsonString); private List<Object> workFlows = configJsonCtx.read("$.Workflow"); this.workFlows.forEach((workFlowObj) -> { try { DocumentContext workFlow = JsonPath.parse(workFlowObj); ........ This is working fine with Nifi 16 and com.jayway.jsonpath.json-path:2.4.0 dependency. But now I am trying to upgrade to nifi 24 and I am getting below error when I do that Any idea what could be the reason and solution for the same. 2024-03-12 05:09:08,815 ERROR [Timer-Driven Process Thread-320] c.o.n.p.o.O9QueueRouting O9QueueRouting[id=8718e632-f033-372b-8f6c-682399fbf9b6] Exception: java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class com.jayway.jsonpath.DocumentContext (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; com.jayway.jsonpath.DocumentContext is in unnamed module of loader org.apache.nifi.nar.NarClassLoader @147e0734) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.lambda$onScheduled$0(O9QueueRouting.java:299) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at com.o9solutions.nifi.processors.o9_custom_processors.O9QueueRouting.onScheduled(O9QueueRouting.java:269) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55) at org.apache.nifi.controller.StandardProcessorNode.lambda$initiateStart$8(StandardProcessorNode.java:1765) at org.apache.nifi.engine.FlowEngine$3.call(FlowEngine.java:123) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Thanks in advnace Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-08-2024
02:20 AM
1 Kudo
Thank you so much @MattWho !
... View more
03-05-2024
08:46 AM
@MattWho - requesting your opinion here (if possible)
... View more
03-04-2024
10:35 PM
1 Kudo
Hello experts, We are using Nifi for data ingestion via http - using 'HandleHttpRequest' processor. Data is getting ingested continuously but once in a while when incoming volume is little high we get 'Service Unavailable - 503' error from nifi end point. Below is the log we see in nifi app logs - 2024-03-04 11:20:25,558 WARN [StandardHttpContextMap-c2ffb1fd-2c09-3a90-0e5e-18384499605f] o.a.nifi.http.StandardHttpContextMap StandardHttpContextMap[id=c2ffb1fd-2c09-3a90-0e5e-18384499605f] Request from 10.13.2.204 timed out; responding with SERVICE_UNAVAILABLE Imp configs : [HandleHttpRequest processor] Container Queue Size : 2000 [StandardHttpContextMap controller service] Maximum Outstanding Requests : 8000 Request Expiration : 3min nifi.web.jetty.threads=200 (in nifi.properties) Maximum Timer Driven Thread Count = 200 Total number of running processors on this env : 4200+ This nifi is running on 8 core machine (ec2 instance) When we look at the CPU usage during the issue we dont see a cpu usage is more than 40% Can anyone help me to understand the issue and the fix? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
02-22-2024
06:55 AM
Thank you very much for the quick reply @cotopaul @MattWho . 2024-02-22 14:44:24,319 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 4290 2024-02-22 14:44:25,086 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootstrap requests on port 44846 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Apache NiFi _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut _ __ ___ __ _(_)___| |_ _ __ _ _ 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | '__/ _ \/ _` | / __| __| '__| | | | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | | | __/ (_| | \__ \ |_| | | |_| | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut |_| \___|\__, |_|___/\__|_| \__, | 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut ==========|___/================|___/= 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut v 2024-02-22 14:44:28,308 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-22 14:44:31,494 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Server start failed: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'standardServiceFacade' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/service/StandardServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseMetadataService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/db/DatabaseMetadataService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.internal.license.FlywayEditionUpgradeRequiredException: Flyway Teams Edition or MySQL upgrade required: MySQL 5.7 is no longer supported by Flyway Community Edition, but still supported by Flyway Teams Edition. 2024-02-22 14:44:32,331 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry ^C root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# mysql --version mysql Ver 14.14 Distrib 5.7.33, for Linux (x86_64) using EditLine wrapper root@nifi12-dev:/usr/independent_nifi/nifi-registry-1.24.0/logs# I see above logs in bootstrap, Is the issue because my mysql version is 5.7? In another server I am running nifi 1.16 registry with this mysql 5.7 version and its working there from 2 years. Thanks Mahendra
... View more
02-21-2024
09:23 AM
Hello experts, I have downloaded and trying to run nifi registry 1.24 with meta data db as mysql and Git flow persistence. Getting below error and unable to start the service - log in text : 2024-02-21 16:11:13,716 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry Runtime Java version: 11.0.14 2024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Starting Apache NiFi Registry... 2024-02-21 16:11:13,725 INFO [main] o.apache.nifi.registry.bootstrap.Command Working Directory: /usr/independent_nifi/nifi-registry-1.24.0 2024-02-21 16:11:13,726 INFO [main] o.apache.nifi.registry.bootstrap.Command Command: java -classpath /usr/independent_ni............................. 2024-02-21 16:11:13,743 INFO [main] o.apache.nifi.registry.bootstrap.Command Launched Apache NiFi Registry with Process ID 19099 2024-02-21 16:11:14,447 INFO [NiFi Registry Bootstrap Command Listener] o.a.n.registry.bootstrap.RunNiFiRegistry Apache NiFi Registry now running and listening for Bootstrap requests on port 37730 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut Apache NiFi _ _ 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut _ __ ___ __ _(_)___| |_ _ __ _ _ 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | '__/ _ \/ _` | / __| __| '__| | | | 2024-02-21 16:11:17,712 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut | | | __/ (_| | \__ \ |_| | | |_| | 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut |_| \___|\__, |_|___/\__|_| \__, | 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut ==========|___/================|___/= 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut v 2024-02-21 16:11:17,713 INFO [NiFi logging handler] org.apache.nifi.registry.StdOut 2024-02-21 16:11:20,550 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Wed Feb 21 16:11:20 UTC 2024 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2024-02-21 16:11:21,763 ERROR [NiFi logging handler] org.apache.nifi.registry.StdErr Server start failed: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accessPolicyResource' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/api/AccessPolicyResource.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'standardServiceFacade' defined in file [/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/classes/org/apache/nifi/registry/web/service/StandardServiceFacade.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'registryService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/service/RegistryService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'databaseMetadataService' defined in URL [jar:file:/usr/independent_nifi/nifi-registry-1.24.0/work/jetty/nifi-registry-web-api-1.24.0.war/webapp/WEB-INF/lib/nifi-registry-framework-1.24.0.jar!/org/apache/nifi/registry/db/DatabaseMetadataService.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flyway' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.flywaydb.core.Flyway]: Factory method 'flyway' threw exception; nested exception is org.flywaydb.core.api.FlywayException: Unable to obtain connection from Flyway DataSource 2024-02-21 16:11:22,745 INFO [main] o.a.n.registry.bootstrap.RunNiFiRegistry NiFi Registry never started. Will not restart NiFi Registry nifi-registry.properties: nifi.registry.web.war.directory=./lib nifi.registry.web.http.host=nifi12-dev.xyz.com nifi.registry.web.http.port=61080 nifi.registry.web.https.host= nifi.registry.web.https.port= nifi.registry.web.https.application.protocols=http/1.1 nifi.registry.web.jetty.working.directory=./work/jetty nifi.registry.web.jetty.threads=200 nifi.registry.web.should.send.server.version=true nifi.registry.security.keystore= nifi.registry.security.keystoreType=jks nifi.registry.security.keystorePasswd= nifi.registry.security.keyPasswd= nifi.registry.security.truststore= nifi.registry.security.truststoreType=jks nifi.registry.security.truststorePasswd= nifi.registry.security.needClientAuth=True nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml nifi.registry.security.authorizer=managed-authorizer nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml nifi.registry.security.identity.provider= nifi.registry.sensitive.props.additional.keys= nifi.registry.providers.configuration.file=./conf/providers.xml nifi.registry.registry.alias.configuration.file=./conf/registry-aliases.xml nifi.registry.extensions.working.directory=./work/extensions nifi.registry.db.directory= nifi.registry.db.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE;useSSL=false # database properties nifi.registry.db.url=jdbc:mysql://localhost:3306/nifi24_registry nifi.registry.db.driver.class=com.mysql.jdbc.Driver nifi.registry.db.driver.directory=/usr/share/java/ nifi.registry.db.username=registry nifi.registry.db.password=PasswordSample1234 nifi.registry.db.maxConnections=5 nifi.registry.db.sql.debug=false Any suggestion / solution is much appreciated. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
02-05-2024
07:13 AM
Thanks for the quick reply @MattWho . We found the issue. It was due to firewall setup for the partition-2 on broker side. When they fixed it started working
... View more
02-05-2024
05:13 AM
1 Kudo
Hello experts, We are facing an issue, where preprod Nifi server is able to read a kafka topic correctly. It is able to read all 3 partitions data. At the same time UAT Nifi server is connecting to same topic with different consumer group id and which is reading only the partition 0 & partition 1 but not partition 2. Screen shot from Lenses abt the same consumer group which is facing issue Any idea what could be the possible reasons? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
11-10-2023
02:55 AM
Hello Experts, We are using Nifi for continuous data ingestion over http port (as rest api) and also we use nifi admin rest api to dynamically start/stop procesors or update variable registry or create process groups etc. We are facing a challenge with these admin rest api's (/nifi-api), i.e when nifi restarted all apis responds very quickly and works fine and when days go on (if no nifi restart in btwn) they nifi rest api's becomes very slow and at some point they stop responding or timing out. And this slowness/not responding issue will resolve only after nifi restart. Since we have real time data ingestion via http port , we cant really restart frequently. Nifi version : 1.16.3 Is there any solution for this problem ? or any suggestion to tackle this would be much appreciated Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
11-03-2023
12:17 AM
1 Kudo
Found the issue, DisrtibtedMapCacheServer was configured with default 1 mb to handle in each cache request. But I was trying to pass a attribute value to pass to wait processor from Notify which was 1.5 mb. After increasing the cache server config from 1mb to 10 mb it worked
... View more