Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 275 | 05-06-2026 09:16 AM | |
| 457 | 05-04-2026 05:20 AM | |
| 335 | 05-01-2026 10:15 AM | |
| 519 | 03-23-2026 05:44 AM | |
| 390 | 02-18-2026 09:59 AM |
03-20-2023
12:58 PM
@apmmahesh You created certificates for each of your NiFi nodes. Base on exception you shared, it appears that you created DNs for those nodes as following? CN=node1, OU=NIFI CN=node2, OU=NIFI CN=node3, OU=NIFI When you have a NiFi cluster, you can manage that cluster via the UI of any one of the connected nodes. So let's say you authenticate via a mutual TLS handshake to node1 using your CN=admin, OU=NIFI certificate you created for yourself and loaded in your browser. What happens next is node1 wants to show you all the data/details from all three nodes and not just node1, so your request to load the NiFi is sent via proxy by node1 to whichever node is the elected cluster coordinator. That cluster coordinator replicates the request on your behalf to all nodes in the cluster. This how the node1 UI would show you details about connected nodes, queued data from other nodes, etc. This means that node1 would need to be authorized to proxy user requests. So typically on first startup secure NiFi will use the configuration in your authorizers.xml to setup these needed default authorization, but your configuration is missing your nodes, so this was not done. Inside your file user-group-provider, you need to also add your NiFi node DNs as users. <userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=admin, OU=NIFI</property>
<property name="Initial User Identity 2">CN=node1, OU=NIFI</property>
<property name="Initial User Identity 3">CN=node2, OU=NIFI</property>
<property name="Initial User Identity 4">CN=node3, OU=NIFI</property>
</userGroupProvider> Then in your file-access-policy-provider you need to add your nodes so that when it generates the authorizations.xml file, the nodes get authorized to the "proxy user requests" policy: <accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">file-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">CN=admin, OU=NIFI</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=node1, OU=NIFI</property>
<property name="Node Identity 2">CN=node2, OU=NIFI</property>
<property name="Node Identity 3">CN=node3, OU=NIFI</property>
<property name="Node Group"></property>
</accessPolicyProvider> NOTE: NiFI will only create the users.xml and authorizations.xml files from the above two providers if they do NOT already exist. Making changes to these providers will not result in changes to existing files. The expectation is that after access for yoru initial admin and your proxy nodes is established that all new authorizations are setup via the NiFi UI which will result in updated to these files. So rename your existing users.xml and authorizations.xml before starting yoru NiFi so new get created. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-17-2023
12:59 PM
@wffger2 A flow definition is a snippet of the components contained with in the Process Group (PG) on which the flow definition was exported. The import of a flow definition is handled differently since a flow definition can be imported over and over to the same NiFi or different NiFis. On import the components will be assigned unique new component UUIDs. So when you create/downloaded your flow definition from DEV and then imported to UAT, the UAT components will have different UUIDs. What you should be doing is install a NiFi-Registry [1] that both your DEV and UAT environments can connect to. This allows you to version control a Process Group (PG) on the your DEV environment and then load that version controlled PG to your UAT environment. While the component UUIDs in UAT will still be different from DEV, both PGs will track back to same version controlled flow stored in the NiFi-Registry. as you make changes in DEV to components in the Version controlled PG, the DEV PG will report that local changes exist. You can commit those local changes as a new version of the PG. At which time the same PG on your UAT env will report a newer version being available which you can change to. You will also have ability to see differences/changes between what is most recent version in NiFi-Registry and what is local to each NiFi from NiFi. [1] https://nifi.apache.org/registry.html If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-17-2023
07:30 AM
@anoop89 This is an unrelated issue to this original thread. Please start a new question. Fell free to @ckumar and @MattWho in your question so we get notified. This issue is related to authorization of your user. Thanks, Matt
... View more
03-14-2023
05:56 AM
@srilakshmi The PublishKafka and PublishKafkaRecord processors do not write any new attributes to the FlowFile when there is a failure. It simply logs the failure to the nifi-app.log and routes the FlowFile to the failure relationship. So on the FlowFile there is no unique error written that can be used for dynamic routing on failure. It could be expensive to write stack traces that come out of Client code to NiFi FlowFiles considering FlowFile attributes/metadata resides in the NiFi heap memory. This may be a topic you want to raise in Apache NiFi jira as a feature/improvement request on these processors to get feedback from Apache NiFi community committers. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-13-2023
12:35 PM
@davehkd I am not sure I am clear on the ask. Are you having issues with your 5 node NiFi cluster? As far as certificates go for NiFi, it really does not matter where you obtain them or if you use self-signed (not recommended) as long as the keystore meets the requirements for NiFi. A NiFi node's keystore must meeting the following requirements: 1. Keystore contains only 1 PrivateKey entry. You can not have multiple PrivateKey Entries in the keystore since NiFi will not know which to use. 2. Keystore PrivateKey entry MUST have Extended Key Usage (EKU) of clientAuth and serverAuth, NiFi nodes communicate with one another and thus will act as clients and servers in the TLS exchange. 3. Keystore PrivateKey entry must contain a DNS entry for the hostname on which the certificate is being used. A NiFi node's truststore contains 1 too many trustedCertEntries. It needs to contain the complete trust chain for any client certificates that will be used to authenticate with NiFi via a mutual TLS handshake. This includes the complete trust chain for each node in yoru cluster. A trust chain consist of every intermediate CA public cert all the way to the root CA public cert. The root CA will have the same owner and issuer. The cacerts file that is included with most java distributions is a truststore containing most public signing authorities intermediate and root CAs. You can obtain a verbose listing of your keystore/truststore using the keytool command found in yoru java install <path to JDK>/bin/keytool -v -list -keystore <keystore or truststore filename> From the output verify following on PrivateKey entry: (DNSName will have your nodes hostname) If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-13-2023
07:22 AM
@dyhiamedjouti It would also be helpful if you shared the full version of Apache NiFi that you installed. The latest versions of NiFi start as secured by default. The Single User "username" and "password" are only output to the log the very first time NiFi is started. Subsequent restarts of NiFi service will not log the username and password again. You can stop your NiFi and run the following command to set your own single user identity provider username and password: $ ./bin/nifi.sh set-single-user-credentials <username> <password> Then when NiFi is up and running, you can use your set username and password to access the UI. Not knowing your username and password has nothing to do with the browser not being able to load the NiFi UI for logging in. When you launch NiFi, this starts the NiFi bootstrap process which will then launch a child process which is the main Nifi process. When this sub process starts, logging will begin in the nifi-app.log. The NiFi Ui will not be accessible until this process has loaded completely and successfully. NiFi will log a few lines that contain UI is available at the following URLs. You'll want to verify you find these log lines and the URLs listed. These are the URLs you will use in your browser to access your NiFi. If you do not see the URLs output in the logs, that means NiFi failed to successfully start. Again the nifi-app.log should provide logging details as to why the sub-process failed during the startup process. Commonly a result of misconfiguration. If you are not seeing a nifi-app.log produced, then check for the nifi-bootstrap.log for any exceptions. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-09-2023
12:54 PM
@RRosa That particular exceptions seems to point an issue with the ldap-provider configuration in your nifi-registry possible related to the manager DN property not being set. Would need to see your nifi-registry.properties and authorizers.xml to provide more context around the above exception. Yes, OIDC is supported in NiFi-Registry 1.19.1. When access in a secured (TLS/SSL Enabled) NiFi-Registry, the UI is displayed as the "anonymous" user. Only "public" buckets will be visible. In order to login via OIDC, you would need to click on the login via OIDC link in the UI. OIDC properties: nifi.registry.security.user.oidc.discovery.url= nifi.registry.security.user.oidc.connect.timeout=5 secs nifi.registry.security.user.oidc.read.timeout=5 secs nifi.registry.security.user.oidc.client.id= nifi.registry.security.user.oidc.client.secret= nifi.registry.security.user.oidc.preferred.jwsalgorithm= nifi.registry.security.user.oidc.additional.scopes= nifi.registry.security.user.oidc.claim.identifying.user= If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-09-2023
11:57 AM
@davehkd Unfortunately, I would need to have access to the nifi-app.log file(s) from each node to dig in deeper. Did you copy the flow.xml.gz, flow.json.gz, users.xml, and authorizations.xml files from NiFi node 1 or 2 to NIFi node 3? These files all need to match in order for a node to join the cluster. 1. The UI of nifi1 or nifi2 shows "2/2" in the status bar just along top of canvas? 2. The UI of nifi3 shows "1/1" in the status bar just along the top of the canvas? If both above are true, this indicates nifi3 is member of a different cluster. Possible result if issue with your ZK or using a different ZK root node (nifi.zookeeper.root.node). Check for any leading or trailing whitespace in your configuration. You may also want to inspect your ZK logs for the connections coming from all three nodes. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Matt
... View more
03-08-2023
07:44 AM
2 Kudos
@GSB If you wanted it always to be two digits, you would need to apply the same if/else NiFi Expression Language (NEL) logic the minute calculations in the working solution provided by @cotopaul ${value:divide(3600):lt(10):ifElse(${value:divide(3600):prepend(0)},${value:divide(3600)})}:${value:divide(60):mod(60):lt(10):ifElse(${value:divide(3600):mod(60):prepend(0)},${value:divide(3600):mod(60)})} A simpler approach would be to use the toDate and Format NEL functions: ${value:toDate('sssss'):format('HH:mm')} I allow 5 's' assuming that max value would be 86,500 seconds (24 hours in a day) and does not matter if value is smaller. This format also allows you to quickly and easily adjust format for example, maybe you don't want to truncate the remaining seconds and use ":format('HH:mm:ss')" instead. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
03-08-2023
06:06 AM
@Bgrilher I not completely clear on your ask here. According to the ValidateJson processor documentation, a FlowFile Attribute is added to FlowFiles that are routed to the "invalid" relationship: You can route this "invalid" relationship via a connection to a logAttribute processor which can write a log line out to the nifi-app.log (default) with what was written to this FlowFile attribute. If you are not actually looking to see it generate log output but just want to see what was written to this FlowFile attribute, you can use NiFi data provenance for this. Data provenance will give you ability to see FlowFile metedata for a FlowFile in all stages throughout the dataflow that FlowFile progressed. You can also view and download the content (if it is still present in a NiFi dataflow or still present in NiFi archive) at by stage of its processing through your dataflow. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more