Member since
07-30-2019
3473
Posts
1642
Kudos Received
1021
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 303 | 06-15-2026 08:08 AM | |
| 507 | 06-03-2026 06:06 PM | |
| 744 | 05-06-2026 09:16 AM | |
| 1714 | 05-04-2026 05:20 AM | |
| 823 | 05-01-2026 10:15 AM |
06-15-2022
11:39 AM
2 Kudos
@Techie123 When you say "Cooperates Linux Machine", are you referring to http://www.colinux.org/ installation? If so, this does not sound like an ideal place to try and run a NiFi instance or NiFi cluster as it appears to isolate that linux to a single core on the host hardware. NiFi can be a resource intensive application, especially when setup as a cluster. When NiFi is started, you can tail the nifi-app.log to observe the complete startup process. NiFi has completed its startup once you see the following lines: 2022-06-15 13:52:52,727 INFO org.apache.nifi.web.server.JettyServer: NiFi has started. The UI is available at the following URLs:
2022-06-15 13:52:52,727 INFO org.apache.nifi.web.server.JettyServer: https://nifi-node1:8443/nifi
2022-06-15 13:52:52,730 INFO org.apache.nifi.BootstrapListener: Successfully initiated communication with Bootstrap At this time the NiFi URL should be reachable. If not, I would be making sure a firewall or some issue with your "Cooperates Linux Machine" setup is not blocking access to the port on which the NiFi is bound. The latter error you shared: o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors implies that your NiFi cluster has been setup to start secured over HTTPS. Secured NiFi nodes must have a configured keystore and truststore. The above error is telling you that node's truststore does not contain the TrustedCertEntry(s) needed to trust the client certificate presented by another node (in this case the elected cluster coordinator node). The NiFi configured keystore must meet these requirements: 1. Contain only ONE PrivateKeyEntry. (this private key is typically unique per node) 2. That Private key must support both ClientAuth and ServerAuth EKUs 3. That PrivateKey must contain a SAN entry for the NiFI host on which it is being used. That SAN must mach the hostname of the host. The NiFi configured truststore must contain the complete trust chain for the nodes private keys. You can use the keytool command to inspect the contents of either the keystore or truststore and verify above requirements are satisfied, keytool -v -list -keystore <keystore or truststore> A certificate (private "PrivateKeyEntry" or public "TrustedCertEntry") will have an owner an issuer. The issuer is the authority for that key. A self-signed certificate will have the same owner and issuer. The truststore needs to have a TrustedCertEntry for each public certificate in the trusts chain. For example: Assume you have a PrivateKey in your keystore with: owner: cn=nifi-node1, ou=nifi
Issuer: cn=intermediate-ca, ou=trust Your Truststore would then need to have a TrustedCertEntry for the public key for: owner: cn=intermediate-ca, ou=trust
issuer: cn=root-ca, ou=trust and: owner: cn=root-ca, ou=trust
issuer: cn=root-ca, ou=trust You know you have the complete trust chain once you reach the public cert where owner and issuer have the same value. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-15-2022
06:27 AM
@Vapper Maybe I am not 100% clear on your question here.. When you say new "flow file" I assume you are talking about the flow.xml.gz file? In NiFi, a "FlowFile" is what you see being queued on connections between processor components and consists of FlowFile content and FlowFile Metadata/Attributes. In a NiFi cluster, all nodes must be running the exact same flow.xml.gz. If the dataflows loaded from the flow.xml.gz do not match the node will not be allowed to join the cluster. In the latest releases of NiFi, Nodes joining a cluster will inherit the cluster dataflow if the local dataflow does not match. Once a cluster is established, a cluster dataflow is elected. That becomes the clusters dataflow. Joining a node to that established cluster will require that joining nodes is using same flow.xm.gz as existing cluster and if not inherit the cluster elected dataflow over the existing flow on the joining node. NiFi will not assume that a joining nodes flow.xml.gz is "newer" or "desired" over the elected cluster dataflow and replace the elected cluster dataflow with that new dataflow. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-14-2022
12:03 PM
@yagoaparecidoti NiFi will treat the identity strings "user.bind" and "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local" as two different users. The identity string being passed to NiFi configured authorizer post successful authentication in yoru current configuration is "user.bind". However, it appears you have configured your initial admin configured in the authorizers.xml configuration file as "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local" which resulted in admin policies being initially setup in the authorizations.xml and users.xml files as this string. Now within the login-identity-providers.xml file you have your ldap-provider configured which is handling your authentication. One of the configurable properties in that ldap-provider can be configured two ways: <property name="Identity Strategy">USE_USERNAME</property> <property name="Identity Strategy">USE_DN</property> USE_USERNAME setting will pass whatever string was entered in the username login window to the authorizer if authentication was successful. USE_DN setting will pass the users DN (post any matching identity mapping pattern modification) to the authorizer. So you are either using the USE_USERNAME option or you have a identity mapping pattern configured in your nifi.properties file that is matching on the full DN returned by USE_DN and trimming just the "user.bind" from that DN before being passed to the Authorizer. Example:
nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),ou=(.*?),dc=lab,dc=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=LOWER
Above PATTERN would match "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local"
and only capture group one ($1) "user.bind" VALUE would be returnedin all LOWERCASE (TRANSFORM). https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties One other important thing to keep in mind here. The file-access-policy and file-user-group-providers in the authorizers.xml file will ONLY build the authorizations.xml and users.xml files if they do NOT already exist. So if you edit the configured initial admin string, what is already configured in those files will not get modified and that configuration change will have not affect. If you remove the existing users.xml and authorizations.xml files before restarting your NiFi if you decide to change your Initial Admin identity string, then on restart a new users.xml and authorizations.xml will be created with your change. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-14-2022
06:11 AM
@Tryfan I think the concept of sending a file to one node is what needs to change here. BY sending to a single node in the NiFi cluster you create a single point of failure. What happens if that one node on your 7 node cluster goes down? You end up with none of the nodes getting that file and outage to your dataflow. A better design is to place this file somewhere that all nodes can pull it from. Maybe it is a commonly mounted file system to all 7 nodes. (getFile processor)? Maybe an external SFTP server (GetSFTP processor)? etc... Then you construct a dataflow where all nodes are retrieving a file independently as needed. Thanks, Matt
... View more
06-14-2022
05:57 AM
1 Kudo
@Techie123 The ExecuteStreamCommand processor is working as designed: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.16.2/org.apache.nifi.processors.standard.ExecuteStreamCommand/index.html Executes an external command on the contents of a flow file, and creates a new flow file with the results of the command. You could route both the "original" and "output stream " relationships via the same outbound connection to a mergeContent processor which can merge the content from both source FlowFiles into a single FlowFile. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.16.2/org.apache.nifi.processors.standard.MergeContent/index.html If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-10-2022
12:43 PM
1 Kudo
@IslamGamal Keep in mind that all the FlowFile attributes for a FlowFile are held in NiFi's JVM heap memory. Creating large attributes on your FlowFiles can quickly eat up a lot of heap memory and affect JVM performance. Thanks, Matt
... View more
06-10-2022
10:49 AM
@Abhishek27Apple Since you are not seeing anything in the NiFi log files... 1. Have you tried using a different web browser like Firefox? 2. Have you tried opening your browser's Developer tools and inspecting the actual rest-api call that was made when you attempt the various actions that fail from with the NiFi UI? 3. Are you going through a proxy or load balancer (is it configured to use sticky sessions?)? 4. Which Browser and version are you using? 5. Have you tried clearing your browser cache? 6. Does same behavior exist using an incognito window in your browser? 7. What java version is your NiFi using? Thank you, Matt
... View more
06-10-2022
08:28 AM
@Mridul_garg Sharing the complete stack trace(s) from the nifi-app.log maybe be helpful in helping you here. When you say you changed the open file limits, what ddi you change it to? what does the output from "ulimit -a" show. Make sure you run this command as the same user that owns your NiFi process. Thanks, Matt
... View more
06-10-2022
07:44 AM
@Abhishek27Apple Something that strikes me as odd in the configuration file authorizers.xml you shared, I don't see the managed-authorizer That provider would look like this and come after the file-user-group-provider and the file-access-policy-provider: <authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer> The nifi.properties file you shared is configured to use this authorizer. However, I would have expected NiFi to fail to start if this authorizer was really missing. Matt
... View more
06-10-2022
07:33 AM
@Abhishek27Apple I don't see in your authorizations for your user where you have granted permissions for the root process group. From the UI you will see an "Operate" panel on the left side of the canvas. Within the "Operate" panel you will see the component currently selected and an icon that looks like a key. With Nothing selected, it defaults to the root process group created when NiFi was started the very first time. Click on that key and add your user to the policies you want your user to have. By default these policies are not set for the admin as they are not needed by an admin. These policies are typically used by developers who will be building dataflows on the canvas. You can find all the access policies in the embedded documentation within NiFi or from the Apache NiFi site: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policies If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more