Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 301 | 12-17-2025 05:55 AM | |
| 362 | 12-15-2025 01:29 PM | |
| 330 | 12-15-2025 06:50 AM | |
| 338 | 12-05-2025 08:25 AM | |
| 585 | 12-03-2025 10:21 AM |
09-27-2021
10:57 AM
1 Kudo
@paygb You are absolutely correct, the Cloudera Manager configuration property within NiFi configurations fro logback.xml does require you to input the entire logbck.xml and then add your modifications. Adding a new logger for the ControllerStatusReportingTask would require adding a logger like this: <logger name="org.apache.nifi.controller.ControllerStatusReportingTask" level="DEBUG"/> You can add this logger in line along with other loggers already in the logback.xml The result would be all logging from this class at a DEBUG level and below would go in to the nifi-app.log. You could optionally add an additional appender (NiFi has three existing appender now for the nifi-app.log, nifi-user.log and nifi-bootstrap.log files). You can copy and modify one of those and the your logger would need to look like this: <logger name="org.apache.nifi.controller.ControllerStatusReportingTask" level="DEBUG" additivity="false">
<appender-ref ref="NEW_APPENDER"/>
</logger> If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-27-2021
10:44 AM
@DSan Not all component (processor, controller service, reporting task, etc.) support the use of the NiFi Expression Language (NEL). Dynamic properties added to the ExtractText processor do not support NEL. For any existing property, you can float your cursor over the question mark next to the Property name to see if it support Full NEL, Partial NEL, or no NEL. When adding a dynamic property to a processor that supports this capability, you will also see icon depicting NEL support: These dynamic properties in the ExtractText use Java Regular Expressions to extract text from the content of the inbound FlowFile. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-24-2021
11:14 AM
@Submar MiNiFi is not capable of receiving data over S2S being sent to it by another NiFi instance running a Remote Process Group (RPG). It is not something that is supported nor recommended even if it was. S2S was built as a way to smartly distribute FlowFiles being sent from one NiFi instance to a cluster of NiFi instances. Since MiNiFi is not a cluster, the overhead associated with fetching S2S details makes no sense. Now you could set up a RPG on your MiNiFi that pulls from a remote outputPort on your NiFi, but again, that is not an ideal or recommended solution. It also would not result in same data landing on every MiNiFi since data is consumed by which ever MiNiFi connects first to the NiFi output port. If the goal here is to send "FlowFiles" (FlowFile metadata/attributes and FlowFile content) from NiFi to MiNiFi, the better option would be to use the ListenHTTP processor on each of your MiNiFi instance and then have a separate PostHTTP processor for each of your 3 MiNiFi instances. Then have the same FlowFiles routed to each of those three postHTTP processors. The PostHTTP processor also has a property "Send as FlowFile" which when set to true will send the entire FlowFile and not just the content to the target MiNiFi. The flows on all 3 of your MiNiFi instances would still be identical with each listening on the same configured listenHTTP port. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-24-2021
10:48 AM
@samarsimha The zookeeper client version used in Apache NiFi versions prior to 1.13 do not support TLS. You'll need to upgrade your NiFi 1.13 from 1.10 to take advantage of the new TLS connectivity to Zookeeper. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-24-2021
10:34 AM
1 Kudo
@Cloud_era Let me see if I understand your use case fully. You are using the listFTP and FetchFTP processor or the GetFTP processor to pull in files from your FTP server. In the ListFTP or GetFTP processor you have configured the path as "/user/Mahesh/test" and have set "Search Recursively" to true so that you pull files found in sub directories including " 202100923 and 20210924". If you are using the GetFTP processor, you should switch to the List and Fetch processor if running on a NiFi cluster. Also keep in mind that the List/Fetch FTP processors are much newer and provide more configuration options/capabilities not found in the legacy GetFTP processor. The GetFTP processor creates a FlowFile Attribute "absolute.path" that contains the full path to the file that is consumed. The ListFTP processor creates a FlowFile Attribute "path" that contains the full path to the file that will consumed by fetchFTP. So you end up in your example with the above attributes set with: /user/Mahesh/test/202100923 /user/Mahesh/test/20210924 using the NiFi Expression Language statement "${absolute.path:getDelimitedField('4','/')}" and above examples, what you would have returned is "test" since that is the 4 delimited field. Field 1 = blank Field 2 = user Field 3 = Mahesh Field 4 = test Field 5 = 20210924 Field 1 is blank because you set your delimiter as "/" and the string starts with a "/". So setting this to "${absolute.path:getDelimitedField('5','/')}" based on your examples would return either "202100923 or 20210924". The problem here is what if your absolute.path values are not always 4 directories deep. for example: 1. /user/Mahesh/test/202100923/subdir1 2. /user/Mahesh/test/20210924/subdir1 3. /user/Mahesh/test/202100923/subdir1/subdir2 4. /user/Mahesh/test/20210924/subdir1/subdir2 Your expression would still return just "202100923 or 20210924". I don't know how or where you are using this folder information later in your dataflow(s), so hard to give recommendations on what to do. But assuming new example i gave above, here are some other NEL options: ${absolute.path:substringAfterLast('/')} would return: 1. subdir1 2. subdir1 3. subdir2 4. subdir2 ${absolute.path:substringAfter('/user/Mahesh/test')} 1. /202100923/subdir1 2. /20210924/subdir1 3. /202100923/subdir1/subdir2 4. /20210924/subdir1/subdir2 If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
09-24-2021
09:45 AM
@nitinchauhan Sharing details around your dataflow, processor being used, etc... may help in getting engagement from the community. The output you shared shows a failed connection to the a NiFI rest-api endpoint (/nifi-api/system-diagnostics) because of connection refused. Details on this processors configurations? What is seen in the logs of the target NiFi which this processor is trying to connect with? Thanks, Matt
... View more
09-24-2021
09:24 AM
1 Kudo
@hegdemahendra That is a possibility. The 'Maximum Timer driven thread count' settings sets a thread pool that is used by the NiFi controller to hand out threads to dataflow components when they execute. The general recommendation is setting this value to 2 to 4 times the number fo cores present on a single NiFi instance (If you are running a NiFi cluster, this is setting is applied per node and not a max across entire cluster). This does not mean that you can not set the thread pool much higher like you have, but you need to do that cautiously and monitor CPU usage over extended periods of time as your dataflows may fluctuate between periods of high and low CPU demand. It is the cycles of high CPU usage that can become problematic. What you have in your scenario is 8 cores trying to service threads (up to 300) for your dataflows, NiFi core level threads (not part of that thread pool), and threads associated to any other services on the host and the OS. So i suspect you have many thread often in CPU wait, waiting on their time on a core. You could also have a scenario where one thread is WAITING on another thread which is also WAITING on something else. So as the system cycles through all these threads you end up with periods of time of what appears to be a hung system Your dataflow components used and how they are configured along with volumes of data play in to the overall CPU usage and length of time a thread is actively executing. Interesting that you stated that all logging stops as well. The fact that all logging stops, makes we wonder if with so many threads, some core thread get left in CPU wait so long they impact logging. Have you tried getting thread dumps from NiFi when it is in this hung state? Examining a series of thread dumps might help pinpoint if you get in to state were you have threads waiting on other threads that are not progressing. You may also want to take a close look at disk IOPS for all NiFi repos which can affect performance with regards to how long a thread takes to complete. Also keep in mind that large dataflows and large volumes of FlowFiles can lead to a need for many open file handles. Make sure your NiFi Service user has access to a LOT of file handles (999,999 fo example). Your dataflows may also be spinning off a lot of processes, so make sure your NiFi service user also has a high process limit. Hope this helps you look for areas to dig in to your issue, Matt
... View more
09-24-2021
08:27 AM
3 Kudos
Objective
This article will share what options a user has when they have lost or forgot the password set for the nifi.sensitive.props.key in the nifi.properties file.
What is the nifi.sensitive.props.key? ----------------------------------------------
The nifi.sensitive.props.key is used to encode any NiFi component added to the NiFi's canvas that has a sensitive property configured with a value (passwords). These "enc{...}" strings are set in the flow.xml.gz that is generated by NIFi, which includes everything dataflow-related set through the NiFi UI. How the nifi.sensitive.props.key gets set has varied over the history of NiFi. In some releases, NiFi would automatically generate a random password when the user did not set the property when NiFi was first launched. The current release requires that a user set this property or NiFi will not start. The nifi.sensitive.props.key is coupled closely with the flow.xml.gz. The flow.xml.gz file can be migrated to other NiFi installations, but in order for that other instance to load that flow.xml.gz, one of the following must be true:
The flow.xml.gz includes no encrypted passwords
The NiFi loading the flow.xml.gz must be configured with the same nifi.sensitive.props.key, as the NiFi where the flow.xml.gz was obtained
What options do I have if can't remember or lost my nifi.sensitive.props.key set password?
----------------------------------------------
NiFi does not store the nifi.sensitive.props.key anywhere unencrypted; however there are two paths available to move beyond this issue so that the flow.xml.gz can be loaded.
OPTION 1: Since the nifi.sensitive.props.key is used to unencrypt the sensitive property values within the flow.xml.gz, a user could edit the flow.xml.gz in order to remove all the configured sensitive property values. Then any nifi.sensitive.props.key can be set and NiFi will be able to start.
Advantages:
All you need is the flow.xml.gz file
Disadvantages:
Clearing all the sensitive properties values will require those values to be re-entered via the NiFi UI after the flow.xml.gz is successfully loaded.
Process:
Use a command-line editor like VIM to locate all occurrences of "enc{.*}" Example: vim flow.xml.gz
:%s/enc{.*}//g
Use the flow.xml.gz in your new NiFi setup with whatever new nifi.senstive.props.key value you want. As you re-enter sensitive property values in your existing dataflows, they will be written in an encrypted format to the flow.xml.gz using the new nifi.senstive.props.key password you configured.
OPTION 2: NiFi provides a toolkit that allows you to change the nifi.sensitive.props.key and re-encrypt the sensitive properties in the flow.xml.gz at the same time.
Advantages:
No need to re-enter all your passwords in the flow.xml.gz
Disadvantages/challenges:
Need access to the following configuration files from original NiFi that goes with the flow.xml.gz
nifi.properties
bootstrap.conf
flow.xml.gz
Process:
Use the NiFi-toolkit "encrypt-config.sh" script. Cloudera includes the nifi-toolkit on each host where NiFi is deployed; the toolkit can also be downloaded from the Apache NiFi site.
Collect copies of the nifi.properties, bootstrap.conf, and flow.xml.gz files from original source working NiFi.
Execute the following toolkit: ./encrypt-config.sh -v -n /<path to copy of>/nifi.properties -o /tmp/nifi.properties.new -b /<path to copy of>/bootstrap.conf -f /<path to copy of>/flow.xml.gz -g /tmp/flow.xml.gz -s <newpassword> -x
After the above command is executed, you will have two new files written out to /tmp:
A new nifi.properties file with the new password (encoded or plaintext depending on bootstrap.conf configuration) set.
new flow.xml.gz with all "enc{...}" re-encoded to using the new nifi.sensitive.props.key provided in the above command.
Use the nifi.senstive.props.* property values (4 of them) from the new nifi.properties file and the flow.xml.gz in your target NiFi.
Thank you,
Matt
... View more
Labels:
09-23-2021
05:31 AM
@paygb CFM 1.0 is based off apache NiFi 1.9 and was released more than 2 years ago. I would recommend upgrading to a much newer release. I am confused about what in the error message leads you to believe it is specific or related to the NiFi Cluster Manager (NCM) which you are correct only existed in the Apache NiFi 0.x versions. With Apache NiFI 1.x, NiFi had some major changes which included the introduction of zero master clustering which did away with the NCM. Zookeeper became a requirement for NiFi clusters so that any node could be elected the role of Cluster Coordinator (CC) at any time. That CC was responsible for the functions previously handled by the NCM. Now the issue you are seeing is specific to the deployment setup done specifically by CM for the NiFi service which restricts access to some directory path thus preventing NiFi from being able to create the directory being used by the S2S protocol used by the Remote Process Groups (RPGs). The following NiFi improvement was done to address this as of the Apache NiFi 1.10 release: https://issues.apache.org/jira/browse/NIFI-6598 CFM 1.1 is based off Apache NiFi 1.11.4 along with some additional fixes (https://docs.cloudera.com/cfm/1.1.0/release-notes/topics/cfm-nifi-patches.html) and would include this improvement. If you found any of the answers provided assisted with your query, please take a moment to login and click on the "Accept as Solution" button below each answer. Thank you, Matt
... View more
09-22-2021
06:52 AM
@wbivp I created a community article long ago that details the Ranger based "NiFi Resource Identifier" policy strings you would use in Ranger to provide various levels of authorization within NiFi. https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 For example, it you want to grant access so that users can access and create components within a PG: /process-groups/<uuid of PG> . Grant Read and Write for your "group" and/or "user" to allow them to view configurations and added components (processors, controller services, child PGs, etc...) within this PG /data/process-groups/<uuid of PG> Granting Read and Write for your "group" and/or "user" to allow user to view and delete content (flowfiles) queued within connection within this PG. Keep in mind that child PGs will inherit the access granted to the parent PG unless explicit policies have been set on the child PG. Hope this helps, Matt
... View more