Member since
07-30-2019
3471
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 143 | 06-03-2026 06:06 PM | |
| 458 | 05-06-2026 09:16 AM | |
| 822 | 05-04-2026 05:20 AM | |
| 493 | 05-01-2026 10:15 AM | |
| 620 | 03-23-2026 05:44 AM |
09-28-2021
12:21 AM
I understand and got the solution to this problem then thank you for your help. Thanks MattWho.
... View more
09-27-2021
10:57 AM
1 Kudo
@paygb You are absolutely correct, the Cloudera Manager configuration property within NiFi configurations fro logback.xml does require you to input the entire logbck.xml and then add your modifications. Adding a new logger for the ControllerStatusReportingTask would require adding a logger like this: <logger name="org.apache.nifi.controller.ControllerStatusReportingTask" level="DEBUG"/> You can add this logger in line along with other loggers already in the logback.xml The result would be all logging from this class at a DEBUG level and below would go in to the nifi-app.log. You could optionally add an additional appender (NiFi has three existing appender now for the nifi-app.log, nifi-user.log and nifi-bootstrap.log files). You can copy and modify one of those and the your logger would need to look like this: <logger name="org.apache.nifi.controller.ControllerStatusReportingTask" level="DEBUG" additivity="false">
<appender-ref ref="NEW_APPENDER"/>
</logger> If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-25-2021
11:52 PM
Hey Matt, Thanks for you advise. I have followed the below approach and worked fine for my issue. Thanks for your help. ${absolute.path:getDelimitedField('5','/')}
... View more
09-24-2021
10:48 AM
@samarsimha The zookeeper client version used in Apache NiFi versions prior to 1.13 do not support TLS. You'll need to upgrade your NiFi 1.13 from 1.10 to take advantage of the new TLS connectivity to Zookeeper. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-24-2021
09:24 AM
1 Kudo
@hegdemahendra That is a possibility. The 'Maximum Timer driven thread count' settings sets a thread pool that is used by the NiFi controller to hand out threads to dataflow components when they execute. The general recommendation is setting this value to 2 to 4 times the number fo cores present on a single NiFi instance (If you are running a NiFi cluster, this is setting is applied per node and not a max across entire cluster). This does not mean that you can not set the thread pool much higher like you have, but you need to do that cautiously and monitor CPU usage over extended periods of time as your dataflows may fluctuate between periods of high and low CPU demand. It is the cycles of high CPU usage that can become problematic. What you have in your scenario is 8 cores trying to service threads (up to 300) for your dataflows, NiFi core level threads (not part of that thread pool), and threads associated to any other services on the host and the OS. So i suspect you have many thread often in CPU wait, waiting on their time on a core. You could also have a scenario where one thread is WAITING on another thread which is also WAITING on something else. So as the system cycles through all these threads you end up with periods of time of what appears to be a hung system Your dataflow components used and how they are configured along with volumes of data play in to the overall CPU usage and length of time a thread is actively executing. Interesting that you stated that all logging stops as well. The fact that all logging stops, makes we wonder if with so many threads, some core thread get left in CPU wait so long they impact logging. Have you tried getting thread dumps from NiFi when it is in this hung state? Examining a series of thread dumps might help pinpoint if you get in to state were you have threads waiting on other threads that are not progressing. You may also want to take a close look at disk IOPS for all NiFi repos which can affect performance with regards to how long a thread takes to complete. Also keep in mind that large dataflows and large volumes of FlowFiles can lead to a need for many open file handles. Make sure your NiFi Service user has access to a LOT of file handles (999,999 fo example). Your dataflows may also be spinning off a lot of processes, so make sure your NiFi service user also has a high process limit. Hope this helps you look for areas to dig in to your issue, Matt
... View more
09-24-2021
08:27 AM
3 Kudos
Objective
This article will share what options a user has when they have lost or forgot the password set for the nifi.sensitive.props.key in the nifi.properties file.
What is the nifi.sensitive.props.key? ----------------------------------------------
The nifi.sensitive.props.key is used to encode any NiFi component added to the NiFi's canvas that has a sensitive property configured with a value (passwords). These "enc{...}" strings are set in the flow.xml.gz that is generated by NIFi, which includes everything dataflow-related set through the NiFi UI. How the nifi.sensitive.props.key gets set has varied over the history of NiFi. In some releases, NiFi would automatically generate a random password when the user did not set the property when NiFi was first launched. The current release requires that a user set this property or NiFi will not start. The nifi.sensitive.props.key is coupled closely with the flow.xml.gz. The flow.xml.gz file can be migrated to other NiFi installations, but in order for that other instance to load that flow.xml.gz, one of the following must be true:
The flow.xml.gz includes no encrypted passwords
The NiFi loading the flow.xml.gz must be configured with the same nifi.sensitive.props.key, as the NiFi where the flow.xml.gz was obtained
What options do I have if can't remember or lost my nifi.sensitive.props.key set password?
----------------------------------------------
NiFi does not store the nifi.sensitive.props.key anywhere unencrypted; however there are two paths available to move beyond this issue so that the flow.xml.gz can be loaded.
OPTION 1: Since the nifi.sensitive.props.key is used to unencrypt the sensitive property values within the flow.xml.gz, a user could edit the flow.xml.gz in order to remove all the configured sensitive property values. Then any nifi.sensitive.props.key can be set and NiFi will be able to start.
Advantages:
All you need is the flow.xml.gz file
Disadvantages:
Clearing all the sensitive properties values will require those values to be re-entered via the NiFi UI after the flow.xml.gz is successfully loaded.
Process:
Use a command-line editor like VIM to locate all occurrences of "enc{.*}" Example: vim flow.xml.gz
:%s/enc{.*}//g
Use the flow.xml.gz in your new NiFi setup with whatever new nifi.senstive.props.key value you want. As you re-enter sensitive property values in your existing dataflows, they will be written in an encrypted format to the flow.xml.gz using the new nifi.senstive.props.key password you configured.
OPTION 2: NiFi provides a toolkit that allows you to change the nifi.sensitive.props.key and re-encrypt the sensitive properties in the flow.xml.gz at the same time.
Advantages:
No need to re-enter all your passwords in the flow.xml.gz
Disadvantages/challenges:
Need access to the following configuration files from original NiFi that goes with the flow.xml.gz
nifi.properties
bootstrap.conf
flow.xml.gz
Process:
Use the NiFi-toolkit "encrypt-config.sh" script. Cloudera includes the nifi-toolkit on each host where NiFi is deployed; the toolkit can also be downloaded from the Apache NiFi site.
Collect copies of the nifi.properties, bootstrap.conf, and flow.xml.gz files from original source working NiFi.
Execute the following toolkit: ./encrypt-config.sh -v -n /<path to copy of>/nifi.properties -o /tmp/nifi.properties.new -b /<path to copy of>/bootstrap.conf -f /<path to copy of>/flow.xml.gz -g /tmp/flow.xml.gz -s <newpassword> -x
After the above command is executed, you will have two new files written out to /tmp:
A new nifi.properties file with the new password (encoded or plaintext depending on bootstrap.conf configuration) set.
new flow.xml.gz with all "enc{...}" re-encoded to using the new nifi.sensitive.props.key provided in the above command.
Use the nifi.senstive.props.* property values (4 of them) from the new nifi.properties file and the flow.xml.gz in your target NiFi.
Thank you,
Matt
... View more
Labels:
09-22-2021
06:52 AM
@wbivp I created a community article long ago that details the Ranger based "NiFi Resource Identifier" policy strings you would use in Ranger to provide various levels of authorization within NiFi. https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 For example, it you want to grant access so that users can access and create components within a PG: /process-groups/<uuid of PG> . Grant Read and Write for your "group" and/or "user" to allow them to view configurations and added components (processors, controller services, child PGs, etc...) within this PG /data/process-groups/<uuid of PG> Granting Read and Write for your "group" and/or "user" to allow user to view and delete content (flowfiles) queued within connection within this PG. Keep in mind that child PGs will inherit the access granted to the parent PG unless explicit policies have been set on the child PG. Hope this helps, Matt
... View more
09-15-2021
12:06 PM
@MattWho Thank you for the thorough reply! A couple things we will need to work out going forward with using Parameters: 1. a way to update site NiFi instances with any updates to the Parameter Contexts 2. a way to update the workflow via the Registry, but not change which Parameter Context the processor groups are assigned to.
... View more
09-02-2021
09:18 AM
@Justee First thing I would do is add a new Attribute on my FlowFile that specifies the year I'd be searching for in the lines contained within the content of that FlowFile. (optional) For example adding an attribute "year" with a value of "1995". In the routeText processor, I'd then be able to use NiFi Expression Language (NEL) in my java regular expression as supported by this processor component: ^\|(.*?)\|(.*?)\|${year}\|(.*?)$ The above java regular expression will match on lines that begin with a pipe "|" followed by a non greedy wildcard match of one or more character until the very next pipe "|", then again for field 2, then for field three I used NEL which resolves to "1995", and then finally i match via wildcard the remainder of the line. Of course you could simply put "1995" in place of "${year}" in the above regex. The routeText processor component configuration would look like this: The result would be two FlowFiles. One FlowFile would be routed to the relationship "1995" (based on property name used) which would have content only containing lines with "1995". The second FlowFile would route to the "unmatched" relationship and would contain all the non-matching lines ( you may to choose to just auto-terminate this relationship if you don't care about these lines). If you found these responses addressed your query, please take a moment to login and click on "Accept as Solution" below each response that helped you. Thank you, Matt
... View more
08-24-2021
10:59 AM
@smartraman This can also be accomplished through a different and more complex configuration of the ReplaceText processor: Using below input content example: {
"TOT_NET_AMT" : "[\"55.00\"]",
"H_OBJECT" : "File",
"H_GROSS_AMNT" : "[\"55.00,58.00\"]",
"TOT_TAX_AMT" : "[9.55]"
} I would set up the replaceText processor as follows: Instead of just searching for those character patterns and replacing them with nothing, I break entire input line-by-line in to a series of capture groups. That way I can omit the capture groups matching the patterns you want removed ([ or [\" or \"] or ]) and then manipulate the capture group containing a possible comma separated list, so that only the last value in that list is returned. I used below java regular expression which results in 5 capture groups: (.*?)([\Q[\E]\\\"|[\Q[\E])(.*?)(\\\"[\Q]\E]|[\Q]\E])(.*?)$ I then used the following Replacement Value in which I used NiFi expression language against the 3rd capture group. If that capture group does not contain any commas, the entire string is returned. With example above and this configuration, you end up with the following new content: {
"TOT_NET_AMT" : "55.00",
"H_OBJECT" : "File",
"H_GROSS_AMNT" : "58.00",
"TOT_TAX_AMT" : "9.55"
} If you found this helped with your latest query, please take a moment to login and click on "Accept as Solution" below this response. Thank you, Matt
... View more