Member since
07-30-2019
3434
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 121 | 01-27-2026 12:46 PM | |
| 533 | 01-13-2026 11:14 AM | |
| 1164 | 01-09-2026 06:58 AM | |
| 970 | 12-17-2025 05:55 AM | |
| 483 | 12-17-2025 05:34 AM |
06-15-2022
11:39 AM
2 Kudos
@Techie123 When you say "Cooperates Linux Machine", are you referring to http://www.colinux.org/ installation? If so, this does not sound like an ideal place to try and run a NiFi instance or NiFi cluster as it appears to isolate that linux to a single core on the host hardware. NiFi can be a resource intensive application, especially when setup as a cluster. When NiFi is started, you can tail the nifi-app.log to observe the complete startup process. NiFi has completed its startup once you see the following lines: 2022-06-15 13:52:52,727 INFO org.apache.nifi.web.server.JettyServer: NiFi has started. The UI is available at the following URLs:
2022-06-15 13:52:52,727 INFO org.apache.nifi.web.server.JettyServer: https://nifi-node1:8443/nifi
2022-06-15 13:52:52,730 INFO org.apache.nifi.BootstrapListener: Successfully initiated communication with Bootstrap At this time the NiFi URL should be reachable. If not, I would be making sure a firewall or some issue with your "Cooperates Linux Machine" setup is not blocking access to the port on which the NiFi is bound. The latter error you shared: o.a.nifi.controller.StandardFlowService Failed to connect to cluster due to: org.apache.nifi.cluster.protocol.ProtocolException: Failed marshalling 'CONNECTION_REQUEST' protocol message due to: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors implies that your NiFi cluster has been setup to start secured over HTTPS. Secured NiFi nodes must have a configured keystore and truststore. The above error is telling you that node's truststore does not contain the TrustedCertEntry(s) needed to trust the client certificate presented by another node (in this case the elected cluster coordinator node). The NiFi configured keystore must meet these requirements: 1. Contain only ONE PrivateKeyEntry. (this private key is typically unique per node) 2. That Private key must support both ClientAuth and ServerAuth EKUs 3. That PrivateKey must contain a SAN entry for the NiFI host on which it is being used. That SAN must mach the hostname of the host. The NiFi configured truststore must contain the complete trust chain for the nodes private keys. You can use the keytool command to inspect the contents of either the keystore or truststore and verify above requirements are satisfied, keytool -v -list -keystore <keystore or truststore> A certificate (private "PrivateKeyEntry" or public "TrustedCertEntry") will have an owner an issuer. The issuer is the authority for that key. A self-signed certificate will have the same owner and issuer. The truststore needs to have a TrustedCertEntry for each public certificate in the trusts chain. For example: Assume you have a PrivateKey in your keystore with: owner: cn=nifi-node1, ou=nifi
Issuer: cn=intermediate-ca, ou=trust Your Truststore would then need to have a TrustedCertEntry for the public key for: owner: cn=intermediate-ca, ou=trust
issuer: cn=root-ca, ou=trust and: owner: cn=root-ca, ou=trust
issuer: cn=root-ca, ou=trust You know you have the complete trust chain once you reach the public cert where owner and issuer have the same value. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-15-2022
06:27 AM
@Vapper Maybe I am not 100% clear on your question here.. When you say new "flow file" I assume you are talking about the flow.xml.gz file? In NiFi, a "FlowFile" is what you see being queued on connections between processor components and consists of FlowFile content and FlowFile Metadata/Attributes. In a NiFi cluster, all nodes must be running the exact same flow.xml.gz. If the dataflows loaded from the flow.xml.gz do not match the node will not be allowed to join the cluster. In the latest releases of NiFi, Nodes joining a cluster will inherit the cluster dataflow if the local dataflow does not match. Once a cluster is established, a cluster dataflow is elected. That becomes the clusters dataflow. Joining a node to that established cluster will require that joining nodes is using same flow.xm.gz as existing cluster and if not inherit the cluster elected dataflow over the existing flow on the joining node. NiFi will not assume that a joining nodes flow.xml.gz is "newer" or "desired" over the elected cluster dataflow and replace the elected cluster dataflow with that new dataflow. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-15-2022
02:38 AM
Thanks for the confirmation @Althotta. You should be logged in to the community to see Accept as Solution option:
I have accepted @MattWho 's and @mnui's responses on your behalf. I hope that's alright with you.
... View more
06-14-2022
08:35 PM
Hi @SAMSAL, thanks a lot for you help. I'm getting data as expected.
... View more
06-12-2022
12:50 AM
@MattWho Thank you for pointing out logback.xml, now am able to generate the users.log.
... View more
06-11-2022
01:40 AM
@MattWho Thanks Matt, I understand what you're saying. in my case it's the other way around, I have clients (MiNiFi) initiating the connections when they want to send or receive data. This helps me deal with remote firewalls not allowing incoming connections and sporadic availability as they're the ones starting the connection. At the moment, with S2S, The MiNiFi's connect periodically (every 10s) to see I there are any flowfiles waiting for them. I would need to figure out how to periodically generate that action to perform the same flow. Example scenarios: 1. Client (MiNiFi side) is running a tail on a file. A new line comes into the file and needs to be sent over to NiFi for processing. 2. NiFi wants to send a file to the remote MiNiFi running a putfile processor. I have both scenarios working right now with S2S with no issues to a bunch of MiNiFi's, it's just that this requires hacky workarounds (scripts to modify the MiNiFi yml) to make this work and I would love to figure out a cleaner solution.
... View more
06-10-2022
02:46 PM
1 Kudo
Hi, regarding the first point I dont think you can do that through regex, this is more of transformation operation and you might need to use JsonJolt Processor to do transromation first to simplify the json and store all id's in an array then use regular expression to extract the ids. It will complicate matter it might affect performance. What kind of SQL are you using ? Do you have any json functions in that sql? If that is the case I would recommend you deffer this process to SQL and utilize sql function to parse json. For example in MS SQL you can use OPENJSON function to transpose json into table and then you can use this table in your query. For the second point, this is because the way the extractText reg expression work it treats like group, so beside the main attribute you will get an indexed attribute for each matched group. You can read more about this here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ExtractText/index.html You can also use this post to see if you can extract one attirbute. I just found about it: https://community.cloudera.com/t5/Support-Questions/Extract-whole-json-flowfilecontent-small-to-single-attribute/m-p/308960
... View more
06-10-2022
08:28 AM
@Mridul_garg Sharing the complete stack trace(s) from the nifi-app.log maybe be helpful in helping you here. When you say you changed the open file limits, what ddi you change it to? what does the output from "ulimit -a" show. Make sure you run this command as the same user that owns your NiFi process. Thanks, Matt
... View more
06-10-2022
08:08 AM
@Elsaa Couple things I would check first. 1. Make sure you do not have two Success relationship connections stacked on top of each other between the "UpdateAttribute" processor and the "CalculateRecordStats" processor. They processor show 4 in and 12 out which makes be think 4 went to three different success connections. You can double click on a connection line to add a bend point that would allow you to click and drag that bend point to see if there is another connection under it. 2. If above is not the issue, take a look at the provenance data for your 8 generated FlowFiles to see at what point in your dataflow the clones happened. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-10-2022
07:51 AM
@Elsaa Is it a standardized filename structure with a standardized date format? Can you share examples? Thanks, Matt
... View more