Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 271 | 05-06-2026 09:16 AM | |
| 454 | 05-04-2026 05:20 AM | |
| 333 | 05-01-2026 10:15 AM | |
| 515 | 03-23-2026 05:44 AM | |
| 389 | 02-18-2026 09:59 AM |
01-04-2024
07:26 AM
@arutkwccu The Apache NiFi 2.0.0-M1 release notes have now been updated with a list of nars that have been moved to the Optional Build Profiles. https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1 Thank you, Matt
... View more
01-04-2024
07:04 AM
@Elnozy Other then you examining a serious of spaced out thread dumps from those two nodes during the long startups, it would be difficult to know the specific reason. But you could look to see if these two nodes have a lot more queued data then the other two nodes. Are their content, flowfile, and provenance repositories a lot larger then the other nodes? As fars as the numerous known startup improvements made over the years, there are no workarounds to them other then getting those improvements through upgrade. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-04-2024
06:48 AM
1 Kudo
@jerome69007 How was your nifi.service created? I am confused as to why it contains both the following lines: ExecStart=/appli/nifi/bin/nifi.sh start
ExecStop=/appli/nifi/bin/nifi.sh stop I would expect the nifi.service to only contain the start command. This explains why on linux startup it is executing the start followed by the stop command. Typically setting NiFi as a service on your linux is performed in a few simple steps: 1. Execute "./nifi.sh install" command. This will setup NiFi as a service allowing to use systemctl command like "systemctl start nifi", "systemctl status nifi", and "systemctl stop nifi". 2. execute "systemctl enable nifi" command to configure nifi service start on system bootup. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-03-2024
08:53 AM
@Elnozy HDF 3.3 (Based off Apache NiFi 1.8) is almost 7 years old at this point in time. CFM 2.1.6 is the latest Cloudera NiFi offering based off Apache NiFi 1.23+ version. There have been many improvements to NiFi over the years including fixes that greatly improve NiFi startup time. Your best option is to upgrade to mitigate many security issues, bugs fixes, and get the many improvements including those that improve startup speed. The difference in startup times is most likely directly associated to the amount of FlowFiles queued on some nodes versus others. But, here are just some examples of fixed issues related to slow startup: NIFI-9382 NIFI-9289 NIFI-7999 If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-03-2024
08:15 AM
@PriyankaMondal What is being logged in the nifi-user.log when the issue happens? Have you tried using your browser's developer tools to look at the data being exchanged in the request with the NiFi cluster? Feels like maybe the site cookies are not being sent to the NiFi node after successful authentication resulting in the exception being seen. Thanks, Matt
... View more
01-03-2024
06:01 AM
@arutkwccu The release notes for minor releases typically include highlights covering any "deprecated" components (This is documented for Apache NiFi 2.0.0-M1 Deprecated Components and Features ) or components moved to "optional build profiles" (such as this specific parquet bundle). This is done help make minor upgrades as seamless and painless as possible. Apache NIFi 2.0.0 is a Major release and as such will have many significant changes including breaking changes as well. As such, it should not be treated the same as minor version upgrade and extra care and evaluation taken during migration from a previous major release version. I agree that is would have been nice if the Apache Community included another confluence page documenting all components to the "Optional Build Profile" with instructions like i added above on how to add them. Glad you are good to go now. Thank you, Matt
... View more
01-03-2024
05:29 AM
@arutkwccu Yes, the added ParquetReader will be available to the record processors that utilize record reader or writer. There is not a lot involved in this process. Simply download these two nar files and place them in the NiFi extensions folder owned by NiFi service user. NiFi takes care of the rest. I added these two nars to my 2.0.0-M1 installation to show you that it works here: If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-02-2024
12:35 PM
@arutkwccu @joseomjr NiFi components evolve from release to release. as part of their evolution the components may have default properties added or removed. That is the case here with invokeHTTP processor. The property "Proxy Type" idd exist as a property in Apache NiFi 1.24 invokeHTTP processor and had a default value of "http". This property was removed in Apache NIFi 2.0.0-M1 invokeHTTP processor. In this specific example: - Earlier version of Apache NIFi 1.x invokeHTTP processor supported configuring proxy properties directly in the processor. Later version of Apache NiFi allowed those same configuration to be abstracted to a StandardProxyConfigurationService controller service (for ease of reuse), but also kept the original direct properties in the processor to avoid upgrade issues for user upgrading within a minor release change. As Apache NiFi 2.0.0-M1 is a major release, clean-up is happening here to remove the deprecated config methods for the newer controller service config method. When a NiFi loads the flow.json.gz from another NiFi it loads the same class and reads in all the configured properties. For any configured property where the current version does not have the same property, the property is added as a custom dynamic property. This may render your component invalid until manual correction is made (in your case deleting this dynamic property). Apache NiFi has always functioned in this way and it is not a bug. It is not possible for NiFi framework to know what "dynamic" properties should or should not be added to component, it simply reads what is found in the flow.json.gz being loaded. Validating your dataflows post any upgrade (not just major version changes like Apache NiFi 1.x to 2.x) is a necessary step. This specific migration is mentioned in the Migrating Deprecated Components and Features guide. You'll know right away if a property is not a default property but rather a "dynamic" custom property when you see it. Only dynamic properties will present the trashcan icon to the right of it allowing you to delete it. Hope this adds clarity here. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Mat
... View more
01-02-2024
12:21 PM
@arutkwccu The parquet components were not "deprecated", they were moved out of the default distribution via https://issues.apache.org/jira/browse/NIFI-12282 Apache limits the max size of the project and at times some nars need to be moved out of the distribution to avoid exceeding that max allowed project download size or components less commonly used may be moved out. Does not mean that these components ate no longer being contributed to or updated with the newer NiFi release versions. It does mean however that you will need to manually download the nifi-parquet-nar and any dependency nar(s) it needs that are not already included in your NiFi distribution. You can download nars directly from maven central. Here is the link to the maven central for the nifi-parquet-nar: https://central.sonatype.com/artifact/org.apache.nifi/nifi-parquet-nar/overview If you look at the "Dependencies" tab, you'll see that there is a dependecy on another nar (nifi-hadoop-libraries-nar). Neither of these nars are inlcuded in the default Apache NiFi 2.0.-M1 release. The simplest way to add these nars to your NiFi is to download them into the <path-toNiFi>/extensions/" folder. NiFi will auto-load nars placed on this folder without any need for a NiFi restart. You can find the nars by clicking on the "versions" tab for each required nar and clciking on "Browse" next to the NiFi version you want the nar for. Here are the direct links for the two nars you need for parquet: nifi-parquet-nar - https://repo1.maven.org/maven2/org/apache/nifi/nifi-parquet-nar/2.0.0-M1/nifi-parquet-nar-2.0.0-M1.nar nifi-hadoop-libraries-nar - https://repo1.maven.org/maven2/org/apache/nifi/nifi-hadoop-libraries-nar/2.0.0-M1/nifi-hadoop-libraries-nar-2.0.0-M1.nar If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-02-2024
09:57 AM
1 Kudo
@elemenop Not all local changes result in a change to the version controlled flow. It is typical for a single NiFi-Registry to be used by multiple NiFi deployments (example would be dev and prod deployments). A flow built in the dev environment may be have different configuration based on that dev environment versus what is used in the same dataflow deployed in the prod environment. The following actions are not considered local changes: stopping/starting processors modifying sensitive property values modifying remote process group URLs updating a processor that was referencing a non-existent controller service to reference an externally available controller service assigning, creating, modifying or deleting parameter contexts creating, modifying or deleting variables Sharing the exact Apache NiFi or Cloudera Flow Management (CFM) version along with the reproduction steps executed in yoru environment would be helpful. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more