Member since
07-30-2019
3436
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 130 | 01-27-2026 12:46 PM | |
| 557 | 01-13-2026 11:14 AM | |
| 1197 | 01-09-2026 06:58 AM | |
| 990 | 12-17-2025 05:55 AM | |
| 493 | 12-17-2025 05:34 AM |
09-24-2021
09:24 AM
1 Kudo
@hegdemahendra That is a possibility. The 'Maximum Timer driven thread count' settings sets a thread pool that is used by the NiFi controller to hand out threads to dataflow components when they execute. The general recommendation is setting this value to 2 to 4 times the number fo cores present on a single NiFi instance (If you are running a NiFi cluster, this is setting is applied per node and not a max across entire cluster). This does not mean that you can not set the thread pool much higher like you have, but you need to do that cautiously and monitor CPU usage over extended periods of time as your dataflows may fluctuate between periods of high and low CPU demand. It is the cycles of high CPU usage that can become problematic. What you have in your scenario is 8 cores trying to service threads (up to 300) for your dataflows, NiFi core level threads (not part of that thread pool), and threads associated to any other services on the host and the OS. So i suspect you have many thread often in CPU wait, waiting on their time on a core. You could also have a scenario where one thread is WAITING on another thread which is also WAITING on something else. So as the system cycles through all these threads you end up with periods of time of what appears to be a hung system Your dataflow components used and how they are configured along with volumes of data play in to the overall CPU usage and length of time a thread is actively executing. Interesting that you stated that all logging stops as well. The fact that all logging stops, makes we wonder if with so many threads, some core thread get left in CPU wait so long they impact logging. Have you tried getting thread dumps from NiFi when it is in this hung state? Examining a series of thread dumps might help pinpoint if you get in to state were you have threads waiting on other threads that are not progressing. You may also want to take a close look at disk IOPS for all NiFi repos which can affect performance with regards to how long a thread takes to complete. Also keep in mind that large dataflows and large volumes of FlowFiles can lead to a need for many open file handles. Make sure your NiFi Service user has access to a LOT of file handles (999,999 fo example). Your dataflows may also be spinning off a lot of processes, so make sure your NiFi service user also has a high process limit. Hope this helps you look for areas to dig in to your issue, Matt
... View more
09-24-2021
08:27 AM
3 Kudos
Objective
This article will share what options a user has when they have lost or forgot the password set for the nifi.sensitive.props.key in the nifi.properties file.
What is the nifi.sensitive.props.key? ----------------------------------------------
The nifi.sensitive.props.key is used to encode any NiFi component added to the NiFi's canvas that has a sensitive property configured with a value (passwords). These "enc{...}" strings are set in the flow.xml.gz that is generated by NIFi, which includes everything dataflow-related set through the NiFi UI. How the nifi.sensitive.props.key gets set has varied over the history of NiFi. In some releases, NiFi would automatically generate a random password when the user did not set the property when NiFi was first launched. The current release requires that a user set this property or NiFi will not start. The nifi.sensitive.props.key is coupled closely with the flow.xml.gz. The flow.xml.gz file can be migrated to other NiFi installations, but in order for that other instance to load that flow.xml.gz, one of the following must be true:
The flow.xml.gz includes no encrypted passwords
The NiFi loading the flow.xml.gz must be configured with the same nifi.sensitive.props.key, as the NiFi where the flow.xml.gz was obtained
What options do I have if can't remember or lost my nifi.sensitive.props.key set password?
----------------------------------------------
NiFi does not store the nifi.sensitive.props.key anywhere unencrypted; however there are two paths available to move beyond this issue so that the flow.xml.gz can be loaded.
OPTION 1: Since the nifi.sensitive.props.key is used to unencrypt the sensitive property values within the flow.xml.gz, a user could edit the flow.xml.gz in order to remove all the configured sensitive property values. Then any nifi.sensitive.props.key can be set and NiFi will be able to start.
Advantages:
All you need is the flow.xml.gz file
Disadvantages:
Clearing all the sensitive properties values will require those values to be re-entered via the NiFi UI after the flow.xml.gz is successfully loaded.
Process:
Use a command-line editor like VIM to locate all occurrences of "enc{.*}" Example: vim flow.xml.gz
:%s/enc{.*}//g
Use the flow.xml.gz in your new NiFi setup with whatever new nifi.senstive.props.key value you want. As you re-enter sensitive property values in your existing dataflows, they will be written in an encrypted format to the flow.xml.gz using the new nifi.senstive.props.key password you configured.
OPTION 2: NiFi provides a toolkit that allows you to change the nifi.sensitive.props.key and re-encrypt the sensitive properties in the flow.xml.gz at the same time.
Advantages:
No need to re-enter all your passwords in the flow.xml.gz
Disadvantages/challenges:
Need access to the following configuration files from original NiFi that goes with the flow.xml.gz
nifi.properties
bootstrap.conf
flow.xml.gz
Process:
Use the NiFi-toolkit "encrypt-config.sh" script. Cloudera includes the nifi-toolkit on each host where NiFi is deployed; the toolkit can also be downloaded from the Apache NiFi site.
Collect copies of the nifi.properties, bootstrap.conf, and flow.xml.gz files from original source working NiFi.
Execute the following toolkit: ./encrypt-config.sh -v -n /<path to copy of>/nifi.properties -o /tmp/nifi.properties.new -b /<path to copy of>/bootstrap.conf -f /<path to copy of>/flow.xml.gz -g /tmp/flow.xml.gz -s <newpassword> -x
After the above command is executed, you will have two new files written out to /tmp:
A new nifi.properties file with the new password (encoded or plaintext depending on bootstrap.conf configuration) set.
new flow.xml.gz with all "enc{...}" re-encoded to using the new nifi.sensitive.props.key provided in the above command.
Use the nifi.senstive.props.* property values (4 of them) from the new nifi.properties file and the flow.xml.gz in your target NiFi.
Thank you,
Matt
... View more
Labels:
09-22-2021
06:52 AM
@wbivp I created a community article long ago that details the Ranger based "NiFi Resource Identifier" policy strings you would use in Ranger to provide various levels of authorization within NiFi. https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 For example, it you want to grant access so that users can access and create components within a PG: /process-groups/<uuid of PG> . Grant Read and Write for your "group" and/or "user" to allow them to view configurations and added components (processors, controller services, child PGs, etc...) within this PG /data/process-groups/<uuid of PG> Granting Read and Write for your "group" and/or "user" to allow user to view and delete content (flowfiles) queued within connection within this PG. Keep in mind that child PGs will inherit the access granted to the parent PG unless explicit policies have been set on the child PG. Hope this helps, Matt
... View more
09-15-2021
11:15 PM
1 Kudo
@Ronman as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.
... View more
09-15-2021
12:06 PM
@MattWho Thank you for the thorough reply! A couple things we will need to work out going forward with using Parameters: 1. a way to update site NiFi instances with any updates to the Parameter Contexts 2. a way to update the workflow via the Registry, but not change which Parameter Context the processor groups are assigned to.
... View more
09-08-2021
11:06 AM
1 Kudo
@patrick_hochste There is nothing built in that can help with this task. The challenges here: 1. A user who can copy and paste a PG can also modify that PG before ti gets started (like change the PG name). 2. The created PG and all the containing components are assigned new unique UUIDs. The log will use these new UUIDs when component is started. 3. How do you distinguish between a PG that is copied and one that is being created new? 4. How do you distinguish between a PG that is being started after it was stopped versus starting a brand new copied PG? 5. A copied PG will get a copy of all components and their current configuration. Since you reference setting up an ew DB, does that mean changes to some components configurations? ------ While there is nothing built in to NiFi that can help you trigger based on a copy, paste, and start of a PG, if you could automate the entire process externally via a script, you could build a controlled and structured copy and paste capability. Keep in mind that anything you can do from directly within the NiFi UI, you can also accomplish via the NiFi Rest API. This flexibility would allow you to automate the copy of an existing PG, the pasting of that PG, modifying of that copied PG, and any other external scripted requirements. If you found the provided response(s) assisted with your query, please take a moment to login and click on "Accept as Solution" below each solution that helped you. Thank you, Matt
... View more
09-02-2021
09:18 AM
@Justee First thing I would do is add a new Attribute on my FlowFile that specifies the year I'd be searching for in the lines contained within the content of that FlowFile. (optional) For example adding an attribute "year" with a value of "1995". In the routeText processor, I'd then be able to use NiFi Expression Language (NEL) in my java regular expression as supported by this processor component: ^\|(.*?)\|(.*?)\|${year}\|(.*?)$ The above java regular expression will match on lines that begin with a pipe "|" followed by a non greedy wildcard match of one or more character until the very next pipe "|", then again for field 2, then for field three I used NEL which resolves to "1995", and then finally i match via wildcard the remainder of the line. Of course you could simply put "1995" in place of "${year}" in the above regex. The routeText processor component configuration would look like this: The result would be two FlowFiles. One FlowFile would be routed to the relationship "1995" (based on property name used) which would have content only containing lines with "1995". The second FlowFile would route to the "unmatched" relationship and would contain all the non-matching lines ( you may to choose to just auto-terminate this relationship if you don't care about these lines). If you found these responses addressed your query, please take a moment to login and click on "Accept as Solution" below each response that helped you. Thank you, Matt
... View more
09-01-2021
12:55 PM
@techNerd After you fetch the two files, you could route the success relationship twice via two different connections. One of those connection continues those two files down you r existing dataflow path. The secondary path could go to a replaceText processor that replaces the entire content of the FlowFile with the filename which is store in a FlowFile attribute named "filename". Editing the content down this secondary path does not affect the content of the FlowFile in the other dataflow path. Now that you have the filename in the content you can use other processors to store that content with the filename wherever you want for future retrieval. or you could merge those flowfile in to single FlowFile that would then contain the list of filenames before you store it. This is just a matter of your specific use case for later consumption of this stored data. Take a look at the putDistributedMapCache processor as an example. If you found this response addressed your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-01-2021
12:41 PM
@ToTew Have you tried using the CompressContent [1] processor? This processor can be configured with "Mode" set to "decompress" and "Compression Format" set to "gzip". [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.14.0/org.apache.nifi.processors.standard.CompressContent/index.html If you found this response addressed your query, please take a moment to login and click on "Accept as Solution" below this post. Than you, Matt
... View more
09-01-2021
12:22 PM
1 Kudo
@SAli12 Please share how you have your MergeContent processor configured. If I am following your query correctly, you have a source FlowFile that you pass through a SplitJson processor which produce a number of output FlowFiles. Later after processing all those individual FlowFile you want to combine them all back in to a single FlowFile so that you can generate only a single FlowFile which gets passed to a putEmail processor. Correct? If that is the case, the SplitJson will create a set of Attributes on each split FlowFile it creates: Within either the MergeContent or MergeRecord processor you have the option to set the "Merge Strategy" to "Defragment". This strategy uses those attributes on the FlowFile to bin files base on the fragment.identifier and makes sure that all files are present before a merge happens***. Note: if all bins are being utilized by different fragment identifiers and another unique identifier comes in, the oldest bin will be processed and sent to failure since it is missing all it fragments. But above should get you the one FlowFile your are looking for instead of multiple FlowFiles with partial merges of merges of your original source file. If you found this response addressed your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more