Member since
12-03-2017
87
Posts
9
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
362 | 07-28-2021 04:42 AM | |
602 | 06-23-2020 10:08 PM | |
3180 | 06-17-2020 11:43 PM | |
1554 | 06-09-2020 01:50 AM | |
1565 | 06-09-2020 12:31 AM |
04-18-2022
11:41 AM
Hello Experts, We have a single node nifi server where we ingest data through http (HandleHttpRequest). When we run a load test for 100K records, after 20-30K records, for few seconds (10-15 secs) flow files will get freeze (stuck at different connection between processors) and after some time they will start moving again. It may happen again after some 30-40K records. Due to this pause between total time taken to 100K will become higher (throughput reduced). Any idea why this is happening? (We have one more server where we do same load test but we have not seen this issue there) Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
04-09-2022
11:02 AM
Hello Team, We have HandleHttpRequest processors in our data ingestion flow which was able to receive upto 350+ messages/sec. Now we have deployed the same flow to a new server and here its not taking more than 150 messages/second. We have check the configuration in both the server all looks similar only, any idea what could be the reason which made it slow? Thanks in advance, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-29-2022
10:58 PM
Hello Experts, I have couple of nodes (not in cluster mode). One of nifi is facing issues - I have HandleHttpRequest processor to receive request and it will work for some time and all of a sudden that will stop listening to requests (though processor is in RUNNING state). Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
02-03-2022
09:13 AM
Hello Experts, I am looking for a regex to configure in "RouteOnContent" processor of Nifi so that based on specific string in flow file content I can route FF. FF content : ● rclone.service - Rclone(file transfer tool) Remote Service
Loaded: loaded (/etc/systemd/system/rclone.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-02-03 09:46:01 UTC; 6h ago
Main PID: 23647 (rclone)
Tasks: 13
Memory: 61.3M
CPU: 5.143s
CGroup: /system.slice/rclone.service
└─23647 /usr/bin/rclone rcd --rc-no-auth --rc-job-expire-duration=60m --cache-tmp-upload-path=/tmp/rclone/upload I want to search for the string "Active: active (running)" in flow file content, if present route to different path. I tried few regex but did not work as expected. Any suggestion would be much appreciated. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
01-19-2022
08:35 AM
Hi @ckumar - thanks for the reply. I think that screenshot lacked info may be due to my permission issue. Same issue has been posted by team mate here - https://community.cloudera.com/t5/Support-Questions/NiFi-Node-showing-2-nodes-and-not-respecting-node-down-fault/td-p/334221 Please take a look at the screenshot attachment there which has more info. And i can login from all 3 nodes ui, and in all nodes it shows up only 1 & 2.
... View more
01-19-2022
07:34 AM
Hello Experts, I have a 3 node nifi cluster with host name as nifi-hatest-01, nifi-hatest-02, nifi-hatest-03. nifi-hatest-01 is primary node at the moment and I can see (in logs) it is receiving heartbeats from 02 & 03. And if I create flows in any one of the 3 nodes it gets sycned to all nodes. But the problem now is the 3rd node (nifi-hatest-03) does not show up in the "cluster" tab of Nifi UI. Any idea what could be the issue? any suggestion would be much appreciated. @MattWho @GangWar @smdas Thanks Mahendra
... View more
Labels:
- Labels:
-
Ambari Blueprints
-
Apache NiFi
01-18-2022
06:58 AM
Hello Experts, I am trying setup a nifi registry with the help of nifi registry docker image from dockerhub. My requirement is to use Git flow persistence and mysql as meta database, but docker image has H2 & Flowpersistence. So I am updating provider.xml for Git and nifi-registry.properties for Mysql db. Git Flowpersistence looks fine but once I restart docker container, nifi-registry.properties is getting updated back to H2 again from MySQL (which I have updated) I am updating and putting below section for MySQL meta db. # database properties nifi.registry.db.directory= nifi.registry.db.driver.class=com.mysql.cj.jdbc.Driver nifi.registry.db.driver.directory=/opt/nifi-registry/driver nifi.registry.db.maxConnections=5 nifi.registry.db.password=P*********** nifi.registry.db.sql.debug=false nifi.registry.db.url=jdbc:mysql://<my_ip>:3306/registry_patch nifi.registry.db.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE;useSSL=false nifi.registry.db.username=registry But after container restart, nifi-registry.properties file getting automatically updated as below : # database properties nifi.registry.db.directory= nifi.registry.db.driver.class=org.h2.Driver nifi.registry.db.driver.directory= nifi.registry.db.maxConnections=5 nifi.registry.db.password=nifireg nifi.registry.db.sql.debug=false nifi.registry.db.url=jdbc:h2:./database/nifi-registry-primary;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE nifi.registry.db.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE;useSSL=false nifi.registry.db.username=nifireg Any idea what is the reason and how can avoid this automatica update ?
... View more
- Tags:
- NiFi
- NiFi Registry
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
11-22-2021
03:18 AM
Hello experts. I have implemented a custom log appender and created jar of that and placed in Nifi "lib" directory. This jar has elastic search related 2 dependencies. Now when I am placing this jar in "lib" directory, nifi is failing to start. This Elastic search has jna library inside it and also one of the nar file of nifi in lib directory also has jna of different version so looks like that is causing the issue. Any way to overcome this problem ? What is the right way to add custom appender to nifi logger ? Thanks in advance, Mahedra
... View more
- Tags:
- NiFi
Labels:
- Labels:
-
Apache NiFi
11-18-2021
02:40 AM
Below is the app log - 2021-11-18 16:07:13,819 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.loadExtensions(StandardExtensionDiscoveringManager.java:156) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:131) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:117) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1042) at org.apache.nifi.NiFi.<init>(NiFi.java:158) at org.apache.nifi.NiFi.<init>(NiFi.java:72) at org.apache.nifi.NiFi.main(NiFi.java:301) Caused by: java.lang.NoSuchFieldError: SIZE at com.sun.jna.platform.win32.WinBase.<clinit>(WinBase.java:52) at com.sun.jna.platform.win32.Kernel32Util.getComputerName(Kernel32Util.java:60) at org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog.<init>(ConsumeWindowsEventLog.java:178) at org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog.<init>(ConsumeWindowsEventLog.java:164) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 9 common frames omitted 2021-11-18 16:07:13,820 INFO [Thread-19] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2021-11-18 16:07:13,820 INFO [Thread-19] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
... View more
11-15-2021
10:09 AM
I tried to change the package name of these 2 dependencies with the help of maven shade plugin, shade has places these dependencies classes in new package. Still whenever I add any of these dependenies to maven/jar nifi is failing to start !!!
... View more
11-12-2021
07:49 AM
Hello Experts, I have implemented a custom log appender which inserts log messages to ElasticSearch. Custom appender class works fine when it is run independently as executable jar. But when I add this custom appender jar file to nifi "lib" directory, nifi stopped working - its not able to start. I have below 2 dependencies in my appender jar - <dependency> <groupId> org.elasticsearch </groupId> <artifactId> elasticsearch </artifactId> <version> 7.15.0 </version> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>7.15.0</version> </dependency> And Nifi has multiple processors related to ElasticSearch so the nar files of these processors also have same classes which are present in above 2 dependencies. This looks to be the issue. I need these ElasticSearch dependencies classes at bootstrap level for logger to get initialised. But the nar files get loaded at later point, so I cant depend on the classes which are part of nar files. What is the better approach to solve this issue ? Any suggestion would be much appreciated. Thanks Mahendra
... View more
- Tags:
- ElasticSearch
- NiFi
Labels:
- Labels:
-
Apache NiFi
11-09-2021
10:30 AM
Hello, I have built a kafka cluster with 2was ssl security, so have keystore and trustore files for client to use. These trustore.jks and keystore.jks files works fine with Kafka command line tool - able to produce and consume kafka messages. Also used keytool list option to make sure that password works fine [keytool -list -v -keystore kafka.client.keystore.jks -- this works fine with the password] But the issue is when I try to use the same in Nifi kafka Publish or Consume processors. When I create SSL context controller service and set the certs and password with type 'JKS' it complains that password or type is invalid.. Have attached screenshots of the error in nifi controller service config. Not sure why only NiFi is unable to use when kafka command line tool is able to use it successfully. Also keytool works with the same password. Any suggestion/hint would be much appreciated. Thanks in advance. - Mahendra
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
11-03-2021
08:59 AM
1 Kudo
Hello Experts, We are using Apache Nifi with Git backed nifi registry. Nifi registry integrated with Git repo master branch in development environment, so when we make changes to the flow and commits to the registry it get commits to master branch of git repo. Now we are facing problem when we need to have monthly release and also in parallel continue development. On a specific day we clone master branch for monthly release and we take that for QA. In case if we find any issue in QA phase we dont have right automated way to fix the only issue as master branch which is integrated with dev nifi will have few more extra commits which are not supposed to be part of current monthly release. In case normal git code like java or other language code, we do cherry pick merge only code fix to monthly from master branch, but looking for right way to handle the same in Nifi registry case. Any suggestion/idea would be appreciated Thanks Mahendra
... View more
- Tags:
- NiFi
- NiFi Registry
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
10-24-2021
09:49 PM
Hello experts, I have implemented a custom log appender to store nifi app log to MongoDB. For this I have created a jar with custom appender code and mongodb driver etc. And I am placing this jar inside nifi "lib" directory and all works fine. Now I am trying to add one more custom appender for ElasticSearch, but when I add below elastic search jars to my appender jar and place it in "lib" directory nifi is failing to start. <!-- https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-high-level-client --> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>7.14.2</version> </dependency> <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch --> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>7.14.2</version> </dependency> I am not able to see any related errors in app or bootstrap log file. Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
09-20-2021
05:26 AM
Hello, We are running single node nifi in 8 core machine and configured 'Maxmum Timer driven thread count' as 300. Once in a while Nifi get totally stuck (wont accept flowfiles) and also no app log will generate till we restart nifi manually. So wondering if its due to nifi thread starvation... In below article 'NiFi Thread starvation' section its mention that system will stuck in resource competition cycle. https://community.cloudera.com/t5/Community-Articles/NiFi-HDF-Dataflow-Optimization-Part-2-of-2/ta-p/245113 So any idea if nifi needs more threads than system available core then will there be any chance of nifi to get stuck ? @Wynner Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
09-17-2021
04:16 AM
You can write a simple custom processor which uses local caching (like Caffeine cache implementation ) with cache expiry time as ur window time and build the logic. May be you can store words in cache and keep updating the count or something like that
... View more
09-17-2021
03:51 AM
Hello Experts, We had an incident where all of a sudden nifi completely stopped accepting any request over http listener processors. And it started working only after nifi restart. What could be the possible reasons ? In metrics I can see the heap usage is not crossing 85%. We are running single node nifi in 8 core machine and configured 'Maxmum Timer driven thread count' as 300 Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
08-08-2021
11:51 PM
@Green_ - Can I have a custom appender configured in logback.xml to collect logs and put that in MongoDB ? would that be possible with nifi logs ?
... View more
08-04-2021
10:43 PM
@Green_ Filtering means I may need to ignore some INFO level logs and consider only DEBUG or ERROR level logs... and I dont have to be using native nifi processors only.. but right way of doing is importtant. Please suggest if you see an other way
... View more
08-04-2021
09:36 AM
Hello Experts, I have a requirement where nifi flows imp log information should be written to MongDB, so that another application can read mongodb and display those interesting logs info on web UI. I am looking for best way to do this. One way I am thinking of is tail nifi log (using tail file processor) and filter for imp log and put into MongoDb using native processor. Please suggest if any other better approaches, thanks in advance. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
07-28-2021
04:42 AM
Update : Below commands worked : git config user.name "myemailuser" git config user.email "myemail@myorg.com" Need to run this inside the cloned repo folder and then start nifi registry.
... View more
07-28-2021
02:17 AM
Hello @alim , Thanks for the detailed explanation. Do you know any solution for this problem - https://community.cloudera.com/t5/Support-Questions/NiFi-registry-commits-templates-to-git-with-default-email/td-p/321648 Thanks Mahendra
... View more
07-27-2021
10:55 PM
Hello Experts, I have configured Git as FlowPersisteceProvider for nifi registry. Tested with test repo and worked fine, but now pointed to my company repo and getting error in push/commit. The error is due to the policy is set in my org git repo, it accepts push/commit from the accout with email as *@myorg.com only. But Even though I configured my credentials (user & PAT) which belongs to my org for Git access from registry, NiFi registry pushes the code to Git with email as "nifiregistry@<hostname>". As its org level git repo ops team cant change this policy, so is there any way I can override this default "nifiregistry@<hostname>" to "my-user@myorg.com" so that git push will not be rejected ? This is how nifi reg is pushing and comitting, Authored and Committed Identity should be changed from default Tried this : https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.2.0/administering-nifi-registry/content/git-user-configuration.html But did not solve Error in registry log : Caused by: org.eclipse.jgit.errors.TransportException: https://o9git.visualstudio.com/CoreDev/_git/nifi-registry-dev: error occurred during unpacking on the remote end: error VS403702: The push was rejected because one or more commits contain author email 'nifiregistry@nifi-test-poc.*****.com' which does not match the policy-specified patterns. at org.eclipse.jgit.transport.BasePackPushConnection.readStatusReport(BasePackPushConnection.java:382) at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:222) at org.eclipse.jgit.transport.TransportHttp$SmartHttpPushConnection.doPush(TransportHttp.java:1112) at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:170) at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:172) at org.eclipse.jgit.transport.Transport.push(Transport.java:1344) at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:169) ... 8 common frames omitted Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
07-08-2021
10:51 PM
@Ven5 - Login to git in browser and create branch called 'develop'. - Login to server(node) where registry is running, got to '/versioned_flows' directory - Clone git repo with newly created specific branch >git clone -b develop https://github.com/myrepo-name.git Thats it. you should see your templates committing to 'develop' branch instead of default master branch.
... View more
06-01-2021
08:29 AM
Hello experts, I have nifi registry with MySQL db as metadata store and FileSystem as Persistence provider. And this registry has 10+ flows with multiple versions. Now I am trying to integrate registry to GIT, that is trying to change persistence provider from FileSystem to GIT. I referred some docs (below urls) and steps to switch persistence provider etc, those all help for fresh GIT integration set and they wont carry/migrate already created buckets/flows/versions from FileSystem to GIT https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#switching-from-other-flow-persistence-provider https://community.cloudera.com/t5/Community-Articles/Storing-Apache-NiFi-Versioned-Flows-in-a-Git-Repository/ta-p/248713 Any help on this would be much appreciated. Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
06-01-2021
07:55 AM
Hi @justenji , have you got any solution for this? I am also looking to migrate the existing bucket/flows of file system provider to GIT.
... View more
06-01-2021
07:45 AM
Thanks for the great explanation Matt, much appreciated.
... View more
05-26-2021
09:26 PM
Hello, I am trying to migrate my Nifi registry from one server to another. Used niFi toolkit CLI for the same. Below are the commands used - On source registry server : registry list-buckets registry list-flows On target registr server: registry create-flow registry sync-flow-versions Flows with versions got migrated properly, but one issues noticed is the version date on target server. Instead of actual date of version it shows migrated date for all versions in target server after migration. Find the attached screnn shot for ref. Is there any way to fix this? Thanks Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
05-24-2021
10:07 AM
Hello, I am trying to use nifi toolkit cli to list the bucket in Nifi registry using below command - /usr/hdf/3.4.1.1-4/nifi-toolkit/bin/cli.sh registry list-buckets -u http://<ip>:<port> But getting below error : /usr/hdf/3. 4.1.1-4/nifi-toolkit$ /usr/hdf/3.4.1.1-4/nifi- toolkit/bin/cli.sh registry list-buckets -u http://<ip>:<port> cli.sh: JAVA_HOME not set; results may vary Exception in thread "main" java.lang. NoClassDefFoundError: javax/annotation/Priority at org.glassfish.jersey.model. internal.ComponentBag. modelFor(ComponentBag.java: 551) at org.glassfish.jersey.model. internal.ComponentBag.lambda$ registerModel$10(ComponentBag. java:477) at org.glassfish.jersey.internal. Errors.process(Errors.java: 316) at org.glassfish.jersey.internal. Errors.process(Errors.java: 298) at org.glassfish.jersey.internal. Errors.process(Errors.java: 229) at org.glassfish.jersey.model. internal.ComponentBag. registerModel(ComponentBag. java:469) at org.glassfish.jersey.model. internal.ComponentBag. register(ComponentBag.java: 380) at org.glassfish.jersey.model. internal.CommonConfig. register(CommonConfig.java: 454) at org.glassfish.jersey.client. ClientConfig$State.register( ClientConfig.java:242) at org.glassfish.jersey.client. ClientConfig.register( ClientConfig.java:619) at org.apache.nifi.registry. client.impl. JerseyNiFiRegistryClient.< init>( JerseyNiFiRegistryClient.java: 110) at org.apache.nifi.registry. client.impl. JerseyNiFiRegistryClient.< init>( JerseyNiFiRegistryClient.java: 54) at org.apache.nifi.registry. client.impl. JerseyNiFiRegistryClient$ Builder.build( JerseyNiFiRegistryClient.java: 226) at org.apache.nifi.toolkit.cli. impl.client. NiFiRegistryClientFactory. createClient( NiFiRegistryClientFactory. java:96) at org.apache.nifi.toolkit.cli. impl.client. NiFiRegistryClientFactory. createClient( NiFiRegistryClientFactory. java:39) at org.apache.nifi.toolkit.cli. impl.command.registry. AbstractNiFiRegistryCommand. doExecute( AbstractNiFiRegistryCommand. java:55) at org.apache.nifi.toolkit.cli. impl.command. AbstractPropertyCommand. execute( AbstractPropertyCommand.java: 74) at org.apache.nifi.toolkit.cli. impl.command.CommandProcessor. processCommand( CommandProcessor.java:251) at org.apache.nifi.toolkit.cli. impl.command.CommandProcessor. processGroupCommand( CommandProcessor.java:232) at org.apache.nifi.toolkit.cli. impl.command.CommandProcessor. process(CommandProcessor.java: 187) at org.apache.nifi.toolkit.cli. CLIMain.runSingleCommand( CLIMain.java:145) at org.apache.nifi.toolkit.cli. CLIMain.main(CLIMain.java:72) Caused by: java.lang. ClassNotFoundException: javax.annotation.Priority at jdk.internal.loader. BuiltinClassLoader.loadClass( java.base@9-internal/ BuiltinClassLoader.java:366) at jdk.internal.loader. ClassLoaders$AppClassLoader. loadClass(java.base@9-internal /ClassLoaders.java:184) at java.lang.ClassLoader. loadClass(java.base@9-internal /ClassLoader.java:419) ... 22 more u This machine has Java 9 openJDK. java -version openjdk version "9-internal" OpenJDK Runtime Environment (build 9-internal+0-2016-04-14- 195246.buildd.src) OpenJDK 64-Bit Server VM (build 9-internal+0-2016-04-14- 195246.buildd.src, mixed mode) Any help would be appreciated. Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
07-07-2020
09:23 AM
Use the date filed extracted in attribute while merging so that merging will happen for like (belongs to same date-time-hour) files. In MergeContent processor set ' Correlation Attribute Name ' field with above populated date attribute name, rest will be taken care. You may need to increase number of buckets in merge processor based on your use case
... View more