Member since
07-30-2019
3387
Posts
1617
Kudos Received
999
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 135 | 11-05-2025 11:01 AM | |
| 380 | 10-20-2025 06:29 AM | |
| 520 | 10-10-2025 08:03 AM | |
| 359 | 10-08-2025 10:52 AM | |
| 396 | 10-08-2025 10:36 AM |
01-07-2025
05:57 AM
1 Kudo
@ravi_tadepally A secured NiFi is always going to require successful authentication and authorization. I assume you are fetching a token because you have configured your secured NiFi to use OIDC based user authentication. But keep in mind that a secured NiFi will always support Mutual TLS based authentication no matter what additional authentication methods have been configured. For Rest-api interactions it is often easier to generate a clientAuth certificate that is trusted by your NiFi's truststore and use that instead for authentication. With mutual TLS based authentication there is no need to fetch any token. You simply include the clientAuth certificate in every rest-api call. You could even handle this task via a NiFi dataflow that utilizes the invokeHTTP processor (configured with a SSL Context Service. Could even just use NiFi's keystore and truststore) to make the rest-api call to fetch Prometheus data and then through that dataflow send it to the desired endpoint. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
01-03-2025
07:37 AM
1 Kudo
@spiker What is currently written in your users.xml and authorizations.xml files in the Nifi conf directory? How many nodes in yoru NiFi cluster? What Apache NiFi version are you using? Thank you, Matt
... View more
01-03-2025
07:16 AM
@spiker With the edits to the authorizers.xml and verified newly generated users.xml and authorizations.xml files, are you still seeing the untrusted proxy WARN in the logs? What do you see now in the logs? Thanks, Matt
... View more
01-03-2025
05:59 AM
@spiker Let's start by focusing on the following two shared log lines: 2025-01-02 10:29:05,167 INFO [NiFi Web Server-52] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 172.24.0.3 [<oncloudtemuser@sossourabh7687gmail.onmicrosoft.com><CN=172.24.0.3, OU=NIFI, O=NIFI, L=HYDRABAD, ST=TELANGANA, C=IN>] GET https://172.24.0.3:8443/nifi-api/flow/current-user
2025-01-02 10:29:05,173 WARN [NiFi Web Server-52] o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed 172.24.0.3 GET https://172.24.0.3:8443/nifi-api/flow/current-user [Untrusted proxy CN=172.24.0.3, OU=NIFI, O=NIFI, L=HYDRABAD, ST=TELANGANA, C=IN] In the first log line we see an authenticated user identity followed by the authenticated node identity for the node receiving access request: <oncloudtemuser@sossourabh7687gmail.onmicrosoft.com><CN=172.24.0.3, OU=NIFI, O=NIFI, L=HYDRABAD, ST=TELANGANA, C=IN> In a NiFi cluster setup nodes will proxy all requests on behalf of the authenticated user to the currently elected NiFi cluster coordinator. This means that all nodes in a NiFi cluster must be authorized to proxy user requests. Establishing the minimum required authorizations needed in a new NiFi setup is handle by the authorizers.xml. In your case, you are using the file-access-policy-provider: <accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">composite-configurable-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">oncloudtemuser@sossourabh7687gmail.onmicrosoft.com</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1"></property>
</accessPolicyProvider> This provider will generated the authorizations.xml file ONLY if it does not already exists. Once it exists all additional authorizations and modifications are made from within the NiFi UI. If you edit the file-access-policy-provider, you'll need to delete the authorizations.xml on all nodes before restarting yoru NiFi. So we see from above that you have you initial admin user identity defined, but have not defined your node(s) identities via "Node Identity 1", Node Identity 2", etc... Before you can define your node identity in the file-access-policy-provider for seeding the node required authorizations, the same case sensitive identical node identity must be returned by the the "composite-confgurable-user-group-provider". Which means that "CN=172.24.0.3, OU=NIFI, O=NIFI, L=HYDRABAD, ST=TELANGANA, C=IN" must be returned by either the "file-user-group-provider" or the "aad-user-group-provider". I believe the file-user-group-provider is where you expect you node identities to be derived from: <userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=172.24.0.3, OU=NIFI, O=NIFI, L=HYDRABAD, ST=TELANGANA, C=IN</property>
<property name="Initial User Identity 2"></property>
</userGroupProvider> Just like the "file-access-policy-provider", the "file-user-group-provider" will ONLY generate the users.xml if it does not already exist. So you will need to delete the users.xml on all your nodes before restarting after editing your authorizers.xml. NOTE: Be mindful of case sensitivity in your user identities. These modifications should get you past your UNTRUSTED PROXY issues when trying to access the NiFi with your authenticated user. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-24-2024
07:02 AM
1 Kudo
@BK84 I would avoid if possible designing dataflows that rely on making rest-api calls to control the processing of data. Any issue that may prevent the success of the rest-api call would have negative impacts on yoru data processing. Based on what you have shared, i'd suggest having your A->B->C dataflow directly connect to your D->E->F dataflow. Since your Python script (C) is responsible for creating the files in the directory from which ListFile (D) checks, why not have your python Script output a FlowFile that contains a list of filenames that it created. A splitContent processor could be used to split that into individual FlowFiles that can be passed directly to a FetchFile so that can be consumed (no need for listFile anymore) for writing to the ElasticSearch. Then lets consider the error handling and how to trigger next run without rest-api. SplitContent->FetchFile->PutElasticSearchJson should be placed in a child process group. That process group should be configured to use FlowFile Concurrency (Single FlowFile per Node) and an Outbound Policy (Batch Output). This means that only 1 FlowFile (FlowFile produced by Python Script that contains list of all files to be fetched) will be allowed to enter this child process group at a time. PutElasticSearchJson has relationships for handling retry, success, failure, and errors. These can be used to handle success as well as report (possibly using PutEmail processor) when processing has issues. On the Success relationship path you could use ModifyBytes to zero out the content and then MergeContent to merge the split FlowFiles back into one FlowFile using "Defragment" strategy. Add a max bin age to force failure of a bin after x amount of time if it does not have all its fragments. SplitContent creates all the required FlowFile attributes needed to support the defragment strategy. Assuming all Files were "success" from PutElasticSearchJson processor, the single defragmented File will be output which you send to an output port in that child process group. Once all FlowFiles in the child process are queued at output ports they will allowed to exit the child process group. Assuming a mergeContent "success". you can use this output FlowFile to trigger the next run (hopefully without using rest-api calls). Since you ddi not share how your data gets started in A and B, I can't really suggest anything here. Bottom line is that avoiding using rest-api calls to control your dataflows leads to faster more efficient processing of your data. Allow your dataflow to handle the error handling, But if you do choose to use rest-api calls, the best way to figure them out is to open developer tools in your browser and then manually perform the interactions needed via the NiFi UI. Through the developer tools you can capture (copy as CURL) the call being made in response to your UI action. Now you are likely using username and password to access your NiFi, but this adds more work to do so through NiFi dataflows (for one thing your username and password would be in plain text in the necessary dataflow to get your user token). So you will want to setup a Keystore and Truststore so authentication is handle via a mutualTLS handshake thus avoiding exposed passwords and need to get a user token. The InvokeHTTP processor can be used to make the rest-api calls and can be configured with and SSLContextService. This is a lot of high level information, but hope it has set you on the path to success. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-24-2024
06:00 AM
1 Kudo
@hegdemahendra As far as your issue goes, it would probably be useful to collect a series of thread dumps (at least spaced 5 minutes apart). Then you would be looking for any threads related to the stopping of components to see if they are progressing or hung. Is it stuck on stopping a specific processor or processor class? Do any of the processors that are being stopped have active threads showing for them? Thank you, Matt
... View more
12-23-2024
07:35 AM
1 Kudo
@hegdemahendra While I don't have an environment right now matching yours to test with, I did want to point out the NiFi Variables are deprecated and removed in NiFi 2.0. The NiFi 1.x branch will cease to be developed further. You need to move away from using NiFi Variables and start using NiFi Parameters instead. Parameters are more useful then Variables since they can be used in any property versus variables only being useable in properties supporting NiFi Expression Language (NEL). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-23-2024
06:50 AM
1 Kudo
@BK84 I suggest starting a new community question for your new query. When you start you r new question, please provide more detail on your ask. Not clear what you mean by "trigger". What is the use case you are trying to implement? Thank you, Matt
... View more
12-23-2024
06:25 AM
1 Kudo
@Krish98 Not enough details here to determine what is going on. Do all your FlowFile that are being merged contain the required FlowFile attributes needed for this to be successful? Name Description fragment.identifier Applicable only if the <Merge Strategy> property is set to Defragment. All FlowFiles with the same value for this attribute will be bundled together. fragment.index Applicable only if the <Merge Strategy> property is set to Defragment. This attribute indicates the order in which the fragments should be assembled. This attribute must be present on all FlowFiles when using the Defragment Merge Strategy and must be a unique (i.e., unique across all FlowFiles that have the same value for the "fragment.identifier" attribute) integer between 0 and the value of the fragment.count attribute. If two or more FlowFiles have the same value for the "fragment.identifier" attribute and the same value for the "fragment.index" attribute, the first FlowFile processed will be accepted and subsequent FlowFiles will not be accepted into the Bin. fragment.count Applicable only if the <Merge Strategy> property is set to Defragment. This attribute indicates how many FlowFiles should be expected in the given bundle. At least one FlowFile must have this attribute in the bundle. If multiple FlowFiles contain the "fragment.count" attribute in a given bundle, all must have the same value. segment.original.filename Applicable only if the <Merge Strategy> property is set to Defragment. This attribute must be present on all FlowFiles with the same value for the fragment.identifier attribute. All FlowFiles in the same bundle must have the same value for this attribute. The value of this attribute will be used for the filename of the completed merged FlowFile. Are the FlowFile attribute values correct? How are the individual FlowFile fragments being produced? I find it odd that you say 1 FlowFile is routed to "merged". This implies the a merge was success. Are you sure the "merged" FlowFile only contains the content of one FlowFile from the set of fragments? Any chance you routed both the "original" and "failure" relationships to the same connection? Can you share the full MergeContent processor configuration (all tabs)? When a FlowFile is routed to "failure", I would expect to see logging in the nifi-app.log related to reason for the failure. What do you see in the nifi-app.log? How many unique bundles are you trying to merge concurrently? I see you have set "Max Num Bins" to 200. So you expect to have 200 unique fragment.identifier bundles to merge at one time? How many FlowFiles typically make up one bundle? I also see you have "Max Bin Age" set to 300 sec (5 mins). Are all FlowFiles with same fragment.identifier making it to the MergeContent processor within the 5 minute of one another? Keep in mind that the MergeContent has the potential to consume a lot of NiFi heap memory depending on how it is configured. FlowFile Attributes/metadata is held in heap memory. FlowFiles that are allocated to MergeContent bins have there FlowFiles held in heap memory. So depending on how many FlowFiles make up a typical bundle, the number of bundles being concurrently handled (200 max), and the amount + size of the individual FlowFile attributes, binned FlowFils can consume a lot of heap memory. Do you encounter any out of memory errors in yoru nifi-app.log (might not be thrown by mergeContent). You must be mindful of this when designing a dataflow that uses MergeContent processors. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
12-16-2024
05:08 AM
1 Kudo
@tono425 It is just a WARNING that can safely be ignored. I found some threads on this here: https://bugs.openjdk.org/browse/JDK-8310626 https://apachenifi.slack.com/archives/C0L9VCD47/p1730736181056549?thread_ts=1730294478.475309&cid=C0L9VCD47 https://www.reddit.com/r/java/comments/17cjajl/why_introduce_a_mandatory_enablenativeaccess/ I suppose you could try adding a new java.arg.<some string/num> to the NiFi bootstrap.conf to "--enable-native-access=ALL-UNNAMED" to see if that stops these Warnings on NiFi startup. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more