Member since
12-03-2017
156
Posts
26
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2078 | 11-03-2023 12:17 AM | |
4147 | 12-12-2022 09:16 PM | |
1562 | 07-14-2022 03:25 AM | |
2389 | 07-28-2021 04:42 AM | |
3325 | 06-23-2020 10:08 PM |
12-17-2021
06:18 AM
Hello @alim, Can you please indicate where I can find the ./conf/providers.xml file ? Also regarding this "Flow snapshot histories are managed as Git commits, meaning only the latest version of Buckets and Flows exist in the Git directory.", is there any way to save all versions of a specific flow in GIT? Thank you!
... View more
11-18-2021
02:40 AM
Below is the app log - 2021-11-18 16:07:13,819 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog could not be instantiated java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.loadExtensions(StandardExtensionDiscoveringManager.java:156) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:131) at org.apache.nifi.nar.StandardExtensionDiscoveringManager.discoverExtensions(StandardExtensionDiscoveringManager.java:117) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1042) at org.apache.nifi.NiFi.<init>(NiFi.java:158) at org.apache.nifi.NiFi.<init>(NiFi.java:72) at org.apache.nifi.NiFi.main(NiFi.java:301) Caused by: java.lang.NoSuchFieldError: SIZE at com.sun.jna.platform.win32.WinBase.<clinit>(WinBase.java:52) at com.sun.jna.platform.win32.Kernel32Util.getComputerName(Kernel32Util.java:60) at org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog.<init>(ConsumeWindowsEventLog.java:178) at org.apache.nifi.processors.windows.event.log.ConsumeWindowsEventLog.<init>(ConsumeWindowsEventLog.java:164) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 9 common frames omitted 2021-11-18 16:07:13,820 INFO [Thread-19] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2021-11-18 16:07:13,820 INFO [Thread-19] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
... View more
09-24-2021
09:24 AM
1 Kudo
@hegdemahendra That is a possibility. The 'Maximum Timer driven thread count' settings sets a thread pool that is used by the NiFi controller to hand out threads to dataflow components when they execute. The general recommendation is setting this value to 2 to 4 times the number fo cores present on a single NiFi instance (If you are running a NiFi cluster, this is setting is applied per node and not a max across entire cluster). This does not mean that you can not set the thread pool much higher like you have, but you need to do that cautiously and monitor CPU usage over extended periods of time as your dataflows may fluctuate between periods of high and low CPU demand. It is the cycles of high CPU usage that can become problematic. What you have in your scenario is 8 cores trying to service threads (up to 300) for your dataflows, NiFi core level threads (not part of that thread pool), and threads associated to any other services on the host and the OS. So i suspect you have many thread often in CPU wait, waiting on their time on a core. You could also have a scenario where one thread is WAITING on another thread which is also WAITING on something else. So as the system cycles through all these threads you end up with periods of time of what appears to be a hung system Your dataflow components used and how they are configured along with volumes of data play in to the overall CPU usage and length of time a thread is actively executing. Interesting that you stated that all logging stops as well. The fact that all logging stops, makes we wonder if with so many threads, some core thread get left in CPU wait so long they impact logging. Have you tried getting thread dumps from NiFi when it is in this hung state? Examining a series of thread dumps might help pinpoint if you get in to state were you have threads waiting on other threads that are not progressing. You may also want to take a close look at disk IOPS for all NiFi repos which can affect performance with regards to how long a thread takes to complete. Also keep in mind that large dataflows and large volumes of FlowFiles can lead to a need for many open file handles. Make sure your NiFi Service user has access to a LOT of file handles (999,999 fo example). Your dataflows may also be spinning off a lot of processes, so make sure your NiFi service user also has a high process limit. Hope this helps you look for areas to dig in to your issue, Matt
... View more
09-17-2021
04:16 AM
You can write a simple custom processor which uses local caching (like Caffeine cache implementation) with cache expiry time as ur window time and build the logic. May be you can store words in cache and keep updating the count or something like that
... View more
08-10-2021
02:11 PM
1 Kudo
@hegdemahendra I have found this article by Pierre V. Where he goes into deeper detail about the logback.xml file. He mentions something that might be relevant to what you're looking for - the following two passages are what caught my eye: ``` "We can also define new appenders in the log configuration file and change it according to our needs. In particular, we could be interested by the SMTP Appender that can send logs via emails based on quite a large set of conditions. Full documentation here." "Obviously you can also configure this configuration file so that NiFi log files integrate with your existing systems. An idea could be to configure a Syslog appender to also redirect the logs to an external system." ``` I myself have never done something like this, but it sounds like a step in the right direction for directly writing logs from nifi to mongo.
... View more
07-28-2021
04:42 AM
Update : Below commands worked : git config user.name "myemailuser" git config user.email "myemail@myorg.com" Need to run this inside the cloned repo folder and then start nifi registry.
... View more
06-02-2021
06:22 AM
1 Kudo
@hegdemahendra The NiFi CLI toolkit [1] can help here to an extent. This toolkit provides the following NiFi-Registry capabilities: registry current-user
registry list-buckets
registry create-bucket
registry delete-bucket
registry list-flows
registry create-flow
registry delete-flow
registry list-flow-versions
registry export-flow-version
registry import-flow-version
registry sync-flow-versions
registry transfer-flow-version
registry diff-flow-versions
registry upload-bundle
registry upload-bundles
registry list-bundle-groups
registry list-bundle-artifacts
registry list-bundle-versions
registry download-bundle
registry get-bundle-checksum
registry list-extension-tags
registry list-extensions
registry list-users
registry create-user
registry update-user
registry list-user-groups
registry create-user-group
registry update-user-group
registry get-policy
registry update-policy
registry update-bucket-policy You can get a description of each by executing: <path to>/cli.sh registry sync-flow-versions -h Since you are changing FlowPersistence providers and not trying to sync flows to a new NiFi-Registry, You really can't use the above "sync-flow-versions" function. Plus, I really don't see it even in that scenario being able to accomplish your goal because you would end up with new flow ids. When you create a bucket in NiFi-Registry it is assigned a bucket if (random uuid). When you version control a Process Group (PG) in NiFi, you choose an existing bucket and it first creates a new flow id (Random UUID assigned to the flow). Then the initial version 1 of that PG flow is created and assigned to that flow id in the NiFi-Registry. Since you cannot force the flow id assigned UUID, syncing flows from registry 1 to registry 2, would not track to your version controlled PGs in your NiFI because of change in flow id. In your scenario, you would need to export all your flows (version by version and it is important you keep rack of the version fo the flow you extract). So for a flow with ID XYZ you may have 6 versions. This means you would use: registry export-flow-version I'd suggest naming the produced json file using source flow id and flow version like XYZ_v1.json, XYZ_v2.json, etc... Example: ./cli.sh registry export-flow-version -ot json -u http://<nifi-registry hostname>:<port>/ -f c97fd570-e2ef-4001-98c9-8810244b6015 -fv 1 -o /tmp/c97fd570-e2ef-4001-98c9-8810244b6015_ver1.json You should then save off your original DB. Delete all existing flows so all you have are your original buckets Then you would need to take all these exported flows and import them back in to registry after switching to your new persistence provider. Now keep in mind before importing each flow version you must first create a new flow within the correct still existing buckets. Keep track of these newly assigned flow ids and which original flow id you are importing in to them (very important) Then you MUST import each new flow in exact version 1 to version x order. If you import version 5 of flow XYZ first it will become version 1 within that new flow Id. The version persisted in the output json is not used when importing, it is assigned the next incremental version in the new flow id. Once you are done here you have a bunch of new flow ids with all your versions imported. Now you need to go edit your flow.xml.gz in NiFi. For every version controlled PG in that flow.xml.gz you will find a section that looks like this: <versionControlInformation>
<registryId>912e8161-0176-1000-ffff-ffff98135aca</registryId>
<bucketId>0cab84ff-399b-4113-9767-687e8e33e48a</bucketId>
<bucketName>bucket-name</bucketName>
<flowId>136b3ba8-bc6f-46dd-afe5-235a80ef8cfe</flowId>
<flowName>flow-name</flowName>
<flowDescription/>
<version>5</version>
</versionControlInformation> Everything here should remain the same except fro the change in "flowId" This would allow you to do a global search and replace on "<flowId>original id</flowId>" to "<flowId>new id</flowId>". Make sure you stop all NiFi nodes, put same modified flow.xml.gz on all nodes (backup original), and start NiFi nodes again. Your PGs should now be tracking to your new flows imported in your registry now backed by the gitFlowPersistenceProvider. [1] https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI Sorry there is no automated path for this. If you found this addressed your query, please take a moment to login and click "Accept" on those solutions which assisted you. Thanks, Matt
... View more
06-01-2021
07:45 AM
Thanks for the great explanation Matt, much appreciated.
... View more
05-24-2021
11:58 AM
1 Kudo
Use JDK 8 or JDK 11, JDK 9 is not supported JDK 9 issues https://github.com/graphhopper/graphhopper/issues/1391
... View more
06-23-2020
10:08 PM
2 Kudos
@AJ121 - Please try this : ${datestring:toDate("yyyyMMdd","GMT"):format("yyyy-MM-dd")} Please accept if it helped/solved your problem.
... View more