Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 296 | 05-06-2026 09:16 AM | |
| 483 | 05-04-2026 05:20 AM | |
| 354 | 05-01-2026 10:15 AM | |
| 522 | 03-23-2026 05:44 AM | |
| 391 | 02-18-2026 09:59 AM |
12-05-2022
11:19 AM
1 Kudo
@dreaminz You can create variables on a process group, those variables are then only available to that process group (scope) on which they were created. NiFi documentation on Variables: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables Variable shave been deprecated in favor of Parameter Contexts: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#parameter-contexts You can create a single parameter context that you add parameters to and then associate the parameter context with multiple process groups. This will allow you to update a parameter in one parameter context and effectively update your flows in multiple process groups. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
12-05-2022
11:09 AM
@grb Your QueryDatabaseTable processor is failing because the dependent controller service is not yet enabled. It appears that controller services is still trying to enable (enabling) because the SQLServerDriver you have configured in that controller service is not compatible with the Java JDK version you are using to run NiFi. What version of NiFi are you using? What version Java is your NiFi using? I recommend updating your Java version to the most recent version of Java JDK 8 or Java JDK 11 (Version 11 only supported in NiFi versions 1.10+). Otherwise, you'll need to find an older version of your SQ driver. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
12-05-2022
10:56 AM
@ajignacio That was a big jump from version 1.9.x to 1.16.x of NiFi. NiFi's data provenance stores, for a configurable amount of time, information about NiFi FlowFiles as they traverse the various processors in your dataflow(s). Over the releases of NiFi both improvements and new implementations of provenance have been introduced. The original version of provenance was org.apache.nifi.provenance.PersistentProvenanceRepository which has since been deprecated in favor of a better performing provider class org.apache.nifi.provenance.WriteAheadProvenanceRepository which is the new default. The following properties from the nifi.properties file are used to configure the provenance repository: nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.directory.default=./provenance_repository
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.max.storage.size=10 GB. (use to be 1 GB)
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.index.shard.size=100 MB
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.warm.cache.frequency= For details on these properties, here is Apache NiFi documentation section: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#provenance-repository The good news is that data provenance retention has no direct relationship to the active FlowFiles traversing your dataflow(s) currently. This means that you can shutdown your NiFi, purge the contents of the current <path to>/provenance_repository directory, adjust the configuration properties as you want, and then restart your NiFi. NiFi will build a new provenance repository on startup. Considering that NiFi only provides limited configurable space (1GB original default to 10GB current default) and age (30 days) as the defaults, you would not be losing much if you were to reset. I am also concerned that the path in the error suggests you also created your original provenance_repository within a subdirectory of the FlowFile_repository which I would not recommend. I would strongly suggest not writing the contents of any one of the four NiFi repositories within each other. Considering the flowfile_repository and content_repository are the two most important repositories for tracking your actively being processed FlowFiles in your dataflow(s), I suggest these each be on their own path and reside on dedicated disk backed by RAID to avoid data loss in the event of a disk failure. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
12-05-2022
10:25 AM
@Sinchan You'll want to inspect the configuration of the the following properties in the nifi.properties configuration file: When you configure a secure NiFi configuration, these properties must be configured. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
12-05-2022
09:16 AM
@Vinylal You can download the Cloudera Manager installer from the following Cloudera page: https://www.cloudera.com/downloads/cdp-private-cloud.html You'll need Cloudera username and password in order to access downloads from Cloudera. If you account with Cloudera and don't know your credentials, you can reach out to your Cloudera account representative. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
11-28-2022
12:01 PM
@Mohamed_Shaaban I recommend starting a new community question with the details specific to your setup. This allows the community to address/assist with your specific setup versus comparing your issue to what was shared in this post. Thanks, Matt
... View more
11-22-2022
01:28 PM
2 Kudos
@drewski7 New processors are created within the community all the time and the documentation for processors should include resource considerations for CPU usage and Memory usage. Just because a processor list memory as a resource consideration, that impact is often a byproduct of how that processor has been configured. you can refer to the imbedded documentation in your installed NiFi instance or you can right click on a processor added to the canvas and select "view usage" from the displayed context menu to go directly to that components embedded documentation page. But processors like ReplaceText, SplitText, SplitContent, SplitJson,... would be examples. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
11-08-2022
11:55 AM
@D5ha Your issue is a mutual TLS handshake issue and really has nothing specific to do with NiFi itself. There are a lot of resources on the web for creating certificates. There are even free services like Tinycert you can use to generate valid certificate meeting the requirements I shared in my last response. Providing guidance on how to create certificates does not make much sense since it can be done so many ways: - Self-signed - public CA - Corporate/private CA etc. Your current shared TLS exception is telling you that the IP or Hostname (you have BLUE line through it in yoru image) was not found as a Subject Alternative Name (SAN) in the certificate created for the server side of this handshake which in yoru case happens to also be your NiFi instance. The Site-To-Site-Bulletin-Reporting-Task is acting as the client in this Mutual TLS handshake and the NiFi server S2S destination URL is the server side of this TLS handshake. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
11-08-2022
11:43 AM
1 Kudo
@Bridewin I two things you may want to try.... 1. GetFile processor was deprecated in favor of the newer ListFile --> FetchFile processors. I'd recommend switching to these processors and see if you have the same observations. 2. I'd suggest enabling debug logging for the GetFile processor class to see what additional logging may show. To do this, you would modify the logback.xml file in NiFi's conf directory. Add the below line down in this file where you see similar lines already. <logger name="org.apache.nifi.processors.standard.GetFile" level="DEBUG"/> If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
11-08-2022
11:34 AM
@Jagapriyan I suspect issue with last modified timestamps since missed files have older last modified timestamp than what was already consumed from the target directory that is compounded by the sub-directory structure. My recommendation is switch to using the listing strategy "Tracking Entities" instead. Tracking Entities will keep track of filenames and timestamps so even an older timestamped file will get consumed if its filename is not in the tracked entities list stored in the distributed cache. Let me know is making this change resolves yoru issue. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more