Member since
07-30-2019
2922
Posts
1449
Kudos Received
850
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1 | 05-06-2024 10:40 AM | |
40 | 05-03-2024 08:41 AM | |
144 | 04-26-2024 06:40 AM | |
208 | 04-25-2024 06:16 AM | |
466 | 04-23-2024 05:56 AM |
06-21-2022
08:23 AM
@Drozu - How does disk usage for your NiFi repositories look? - Any ERRORS or WARN logging related writing content? - What do you have set for your "Max Timer Driven Thread cCount" in controller settings (how many cores does the host where you NiFi is running have?)? - Is this a standalone NiFi or a NiFi multi-node cluster (what is the distribution of the FlowFiles on the inbound connections?)? - How has the MergeContent processor been configured? Thanks, Matt
... View more
06-21-2022
08:04 AM
@ThongPham Sounds like you may making a lot of unnecessary rest-api calls that could impact your NiFi's overall performance. Have you maybe looked at using the SiteToSiteBulletingReportingTask? This reporting task will send a FlowFile to a remote input port upon execution of bulletin(s) are produced. That Remote Input Port could then be built into a dataflow that makes notifications via putEmail. So instead of constantly calling the rest-api to see if something happened in the last 5 minutes, the flow will simply send something out when it happens only. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-17-2022
05:56 AM
1 Kudo
@ThongPham There is no such thing as a permanent Bearer token. How long a Bearer token stays valid is set in the provider that issuing that bearer token. In you case the ldap-provider. Also keep in mind that a bearer token is issued by a specific node in the NiFi cluster and can not be used to authenticate with every node in the NiFi cluster. Since a secured NiFi will always attempt mutual TLS authentication first. I suggest you instead you generate and use a client certificate to interact with the NiFi API. Mutual TLS based authentication does not use bearer tokens and the authentication will be successful until that client certificate expires which is configurable when generating the certificate. But generally speaking certificates are often valid for 12 or months. Since there is no bearer token, a client certificate can be used with any node in the cluster. Your other option is to build a flow within your NiFi to get a new bearer token automatically and store that token in maybe a distributedMapCache. Then in your other flow you fetch that bearer token before calling the rest-api endpoint. A failure should loop back to the FetchDistrubutedMapCache just in case you have a scenario where the bearer token expires between fetch and call. Out of curiosity, what rest-api endpoint are you calling every 20 seconds and why? If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-16-2022
12:53 PM
@yagoaparecidoti Authentication is one piece of being able to access NiFi's UI. While the file-user-group-provider and file-access-policy-provider facilitate the automatic creation of the initial admin user identity and setting of Admin needed policies for that user, It is the responsibility of that admin to add additional users to NiFi and add that user to authorization policies. The initial admin user would accomplish this directly form within the NiFi UI. Global menu in upper right corner --> users (add additional user identities here for which you want to setup authorizations) Global menu in upper right corner --> Policies (add users you have added to select NiFi controller policies) From canvas --> operate panel on left side --> key icon to add policies for specific components added to canvas https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#config-users-access-policies Thank you, Matt
... View more
06-15-2022
11:39 AM
2 Kudos
@Techie123 When you say "Cooperates Linux Machine", are you referring to http://www.colinux.org/ installation? If so, this does not sound like an ideal place to try and run a NiFi instance or NiFi cluster as it appears to isolate that linux to a single core on the host hardware. NiFi can be a resource intensive application, especially when setup as a cluster. When NiFi is started, you can tail the nifi-app.log to observe the complete startup process. NiFi has completed its startup once you see the following lines: 2022-06-15 13:52:52,727 INFO org.apache.nifi.web.server.JettyServer: NiFi has started. The UI is available at the following URLs:
2022-06-15 13:52:52,727 INFO org.apache.nifi.web.server.JettyServer: https://nifi-node1:8443/nifi
2022-06-15 13:52:52,730 INFO org.apache.nifi.BootstrapListener: Successfully initiated communication with Bootstrap At this time the NiFi URL should be reachable. If not, I would be making sure a firewall or some issue with your "Cooperates Linux Machine" setup is not blocking access to the port on which the NiFi is bound. The latter error you shared: o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors implies that your NiFi cluster has been setup to start secured over HTTPS. Secured NiFi nodes must have a configured keystore and truststore. The above error is telling you that node's truststore does not contain the TrustedCertEntry(s) needed to trust the client certificate presented by another node (in this case the elected cluster coordinator node). The NiFi configured keystore must meet these requirements: 1. Contain only ONE PrivateKeyEntry. (this private key is typically unique per node) 2. That Private key must support both ClientAuth and ServerAuth EKUs 3. That PrivateKey must contain a SAN entry for the NiFI host on which it is being used. That SAN must mach the hostname of the host. The NiFi configured truststore must contain the complete trust chain for the nodes private keys. You can use the keytool command to inspect the contents of either the keystore or truststore and verify above requirements are satisfied, keytool -v -list -keystore <keystore or truststore> A certificate (private "PrivateKeyEntry" or public "TrustedCertEntry") will have an owner an issuer. The issuer is the authority for that key. A self-signed certificate will have the same owner and issuer. The truststore needs to have a TrustedCertEntry for each public certificate in the trusts chain. For example: Assume you have a PrivateKey in your keystore with: owner: cn=nifi-node1, ou=nifi
Issuer: cn=intermediate-ca, ou=trust Your Truststore would then need to have a TrustedCertEntry for the public key for: owner: cn=intermediate-ca, ou=trust
issuer: cn=root-ca, ou=trust and: owner: cn=root-ca, ou=trust
issuer: cn=root-ca, ou=trust You know you have the complete trust chain once you reach the public cert where owner and issuer have the same value. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-15-2022
06:27 AM
@Vapper Maybe I am not 100% clear on your question here.. When you say new "flow file" I assume you are talking about the flow.xml.gz file? In NiFi, a "FlowFile" is what you see being queued on connections between processor components and consists of FlowFile content and FlowFile Metadata/Attributes. In a NiFi cluster, all nodes must be running the exact same flow.xml.gz. If the dataflows loaded from the flow.xml.gz do not match the node will not be allowed to join the cluster. In the latest releases of NiFi, Nodes joining a cluster will inherit the cluster dataflow if the local dataflow does not match. Once a cluster is established, a cluster dataflow is elected. That becomes the clusters dataflow. Joining a node to that established cluster will require that joining nodes is using same flow.xm.gz as existing cluster and if not inherit the cluster elected dataflow over the existing flow on the joining node. NiFi will not assume that a joining nodes flow.xml.gz is "newer" or "desired" over the elected cluster dataflow and replace the elected cluster dataflow with that new dataflow. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-14-2022
12:03 PM
@yagoaparecidoti NiFi will treat the identity strings "user.bind" and "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local" as two different users. The identity string being passed to NiFi configured authorizer post successful authentication in yoru current configuration is "user.bind". However, it appears you have configured your initial admin configured in the authorizers.xml configuration file as "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local" which resulted in admin policies being initially setup in the authorizations.xml and users.xml files as this string. Now within the login-identity-providers.xml file you have your ldap-provider configured which is handling your authentication. One of the configurable properties in that ldap-provider can be configured two ways: <property name="Identity Strategy">USE_USERNAME</property> <property name="Identity Strategy">USE_DN</property> USE_USERNAME setting will pass whatever string was entered in the username login window to the authorizer if authentication was successful. USE_DN setting will pass the users DN (post any matching identity mapping pattern modification) to the authorizer. So you are either using the USE_USERNAME option or you have a identity mapping pattern configured in your nifi.properties file that is matching on the full DN returned by USE_DN and trimming just the "user.bind" from that DN before being passed to the Authorizer. Example:
nifi.security.identity.mapping.pattern.dn=^cn=(.*?),ou=(.*?),ou=(.*?),dc=lab,dc=(.*?)$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=LOWER
Above PATTERN would match "cn=user.bind,ou=USERS,ou=CLOUDERA,dc=lab,dc=local"
and only capture group one ($1) "user.bind" VALUE would be returnedin all LOWERCASE (TRANSFORM). https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties One other important thing to keep in mind here. The file-access-policy and file-user-group-providers in the authorizers.xml file will ONLY build the authorizations.xml and users.xml files if they do NOT already exist. So if you edit the configured initial admin string, what is already configured in those files will not get modified and that configuration change will have not affect. If you remove the existing users.xml and authorizations.xml files before restarting your NiFi if you decide to change your Initial Admin identity string, then on restart a new users.xml and authorizations.xml will be created with your change. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-14-2022
06:11 AM
@Tryfan I think the concept of sending a file to one node is what needs to change here. BY sending to a single node in the NiFi cluster you create a single point of failure. What happens if that one node on your 7 node cluster goes down? You end up with none of the nodes getting that file and outage to your dataflow. A better design is to place this file somewhere that all nodes can pull it from. Maybe it is a commonly mounted file system to all 7 nodes. (getFile processor)? Maybe an external SFTP server (GetSFTP processor)? etc... Then you construct a dataflow where all nodes are retrieving a file independently as needed. Thanks, Matt
... View more
06-14-2022
05:57 AM
1 Kudo
@Techie123 The ExecuteStreamCommand processor is working as designed: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.16.2/org.apache.nifi.processors.standard.ExecuteStreamCommand/index.html Executes an external command on the contents of a flow file, and creates a new flow file with the results of the command. You could route both the "original" and "output stream " relationships via the same outbound connection to a mergeContent processor which can merge the content from both source FlowFiles into a single FlowFile. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.16.2/org.apache.nifi.processors.standard.MergeContent/index.html If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-10-2022
12:43 PM
1 Kudo
@IslamGamal Keep in mind that all the FlowFile attributes for a FlowFile are held in NiFi's JVM heap memory. Creating large attributes on your FlowFiles can quickly eat up a lot of heap memory and affect JVM performance. Thanks, Matt
... View more