Member since
07-30-2019
3432
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 105 | 01-27-2026 12:46 PM | |
| 511 | 01-13-2026 11:14 AM | |
| 1116 | 01-09-2026 06:58 AM | |
| 951 | 12-17-2025 05:55 AM | |
| 451 | 12-17-2025 05:34 AM |
05-29-2018
04:33 PM
@Paul Hernandez Just to add to the above correct response... The backpressure threshold settings for both size and number of FlowFiles are soft limits. When a processor is eligible to execute/run, it will run that thread to completion. The ListHDFS processor for example will list all FlowFiles newer then the last execution/run recorded state. Even if "Back Pressure Object Threshold" is set to 10000, it will not stop the listHDFS processor from listing 1,000,000 flowfiles in a single execution. Once those 1,000,000 FlowFiles are placed on connection back pressure starts being applied. The listHDFS processor will not be eligible to execute/run again until that threshold drop back below the threshold setting of 10,000. - Back pressure Data Size Threshold" works in a similar manor. Size in NiFi is always a measure of the size of the content associated to a FlowFile and not the actual size of a FlowFile. - Thanks, Matt
... View more
05-29-2018
02:42 PM
2 Kudos
@Raja K There are two Java process associated to a running NiFi instance. The first JVM process is tied to the NiFi bootstrap. It is what you are interacting with when you issue the "start, stop, restart, dump, etc..." During the startup phase, the bootstrap will start the other NiFi JVM process. - During startup of the main NiFi process, there is a lot that goes on. For example: - NiFi must unpack all the nars in the NiFi lib directory to the Nifi work directory and load relevant code in to memory. - NiFi must join the cluster ----- If entire cluster has just been started/restarted, NiFi goes in to an election phase (Cluster needs to elect a cluster coordinator, primary node, a cluster flow). The election will not occur until all nodes have joined (based on nifi.cluster.flow.election.max.candidates=) or until election wait time expires (nifi.cluster.flow.election.max.wait.time=5 mins). If max.candidates is not set, you will always have a min 5 min latency here. - Nodes joining the cluster must then compare their local flow with the cluster flow to make sure they match exactly. - Once the node successfully passes these steps it must start loading flowfiles back in to the flow (these would be flowfiles still queued in connection when NiFi was last stopped/restarted). - Processor components are then scheduled to run based on the cluster flow processor state (start all processors that are in a running state. Start primary node only processors on the elected primary node) - Only then is the cluster as a whole in a state where user should be given access to the UI for interactive purposes. This does not mean that the flows were not already running before theUI was actually available to the user. - There are are things that happening there, but those are the biggest pieces. Tailing the nifi-app/log watching for the "The UI is available at the following URLs:" is the easiest way to see when the UI will be available for the users. Those UIs are based on the following properties from the nifi.properties file: nifi.web.http.host= <-- unsecured nifi
nifi.web.https.host= <-- secured nifi If they are left blank, NiFi will bind to all available interfaces that the JVM finds on the host machine (as your UIs shows). There is no redirection going on here. - Hope this helps explain what is occurring during the startup. The size of your dataflow(s), election process, number of queued FlowFiles being loaded back in to JVM memory (queued connections),etc. will have some impact on startup time. Your output above only has the timestamp for when the UI became available (2018-05-29 06:58:05,832), so it is not clear how much latency you are really seeing between the actual start command that timestamp. First entry in nifi-app.log should include (org.apache.nifi.NiFi Launching NiFi...). - Thank you, Matt
... View more
05-29-2018
02:05 PM
@Mike Wong Not sure what issue you are exactly seeing here based on your screenshots. Your dataflow consists of two processors and I see no queued FlowFiles. Is the GetFile retrieving your 2008.csv file? Based on your configuration it should be retrieving it non stop (keep source file = true). What user owns the running nifi software on this same local machine? (# ps - ef |grep nifi) If you become that user, can you access the file you are trying to consume with NiFi? What do you see in the nifi-app.log when you "start" the GetFile processor? (Any WARN or ERROR logs) - Thanks, Matt
... View more
05-29-2018
01:22 PM
@Henrik Olsen We completely understand how load-balanced redistribution of FlowFiles via a RPG is not the most elegant solution. There was consideration of separating input/output ports in to two different components (local and remote). In addition to the complexity with this, we also have to consider the backward compatibility. What impact with this have on NiFi users upgrading with flows developed with previous versions of NiFi. - Another option being looked at is adding the ability to enable load-balancing within the cluster directly on any connection. This would just be a new configuration option on existing connections with the default just behaving as connections do now. By enabling the load-balancing option, FlowFiles would be load-balanced across all connected nodes behind the scenes automatically. There are still technical hurdle here and no time table for this effort as of now. - Thank you, Matt
... View more
05-29-2018
01:00 PM
@Dilip Namdev The 409 conflict response from NiFi is typical when the service is not available to serve the request. In a NiFi cluster you requests must be replicated to the other node(s). It may be possible that those replication requests failed. It may also be possible that you had a node disconnected. - You should inspect the nifi-user and nifi-app logs at around the time of these failed rest-api calls to see what state you NiFi was in or if there was an issue with a replication request. - Thank you, Matt
... View more
05-23-2018
09:00 PM
@Zack Riesland Your problem is definitely external to NiFi. - Perhaps a NAT issue between your outside and the instance running in EC2. Maybe Security Groups configuration issue? Maybe Network access control list issue? - https://aws.amazon.com/premiumsupport/knowledge-center/instance-vpc-troubleshoot/ - From your external machine you should be able to use command like telnet, openssl, or netcat to verify ability to connect to the NiFi URL endpoint in EC2. - Sorry I won't be much help with EC2 specific issues. You could always start a new community thread asking for help specifically with accessing http endpoints within a EC2 in a VPC. - Thanks, Matt
... View more
05-23-2018
08:26 PM
@Srini K Managing TLS/SSL issued certificates is not a job for NiFi. The NIFi CA is not backed by any type of long time management capability. Is becomes complicated to deploy and manage across multiple NiFi deployments. You are experiencing some of that management difficulty here now. - 1. Every Ambari based NiFi installation sets up its own CA. So it becomes difficult to setup communication between systems where each has a different Unique CA signing their certificates. You are trying to get two NiFi's to talk to one another, but this difficulty would extend to any other external service that NiFi needs to communicate to or receive connections from over TLS/SSL. 2. There is no build in management process to handle certificate renews or notifications of expiration. Your NiFi system may be working fine one day in production and then stop working all together the next because your certs expired. - In a production managed environment, a corporately or external managed CA should be used to issue, sign, and manage all your certificate needs. - Yes, NiFi requires TLS/SSL certificates in order to secure NiFi, but SSL/TLS is not a product of NiFi. - As far as merging the content of your two truststore in to a new truststore... A truststore cannot contain multiple keys with the same alias. Each entry must have a unique alias. The trustedCertEntries in each of your existing truststores have the same alias, so you are going to need to change the alias of one of them. Following commands will extract the trustedCertEntry from each source truststore.jks and put tehem in a new-truststore.jks with new unique alias fro each of those entries: keytool -importkeystore -srckeystore truststore.jks -destkeystore new-truststore.jks -srcalias nifi-cert -destalias nifi-cert1 -srcstorepass ****-deststorepass **** keytool -importkeystore -srckeystore truststore.jks -destkeystore new-truststore.jks -srcalias nifi-cert -destalias nifi-cert2 -srcstorepass ****-deststorepass ****
... View more
05-23-2018
08:04 PM
@Zack Riesland Hope I know NiFi, been working with it almost since the beginning (8+ years now) 🙂 - The only port that needs to be open for users to access the NiFi UI of an unsecured NiFi is which ever port is defined in the nifi.web.http.port= property in the nifi.properties file. - That being said, is your server hosting NIFi multi-homed? The nifi.web.http.host= property is typically configured with the FQDN for your host. When left blank NiFi will bind to every interface on your host. When set it will bind to only the interface which the FQDN is assign to. So maybe you are having issues because NiFi has bound to an interface you cannot reach from other computers on your network? - During startup you will see a line that says: org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs: immediately following that line will be a list of URLs that can be used to access this running NiFi instance. - Some components in NiFi that create their own listening port (example: ListenHTTP) would have a unique port that would need to be opened before they could be accessed by external systems as well - Thanks, Matt
... View more
05-23-2018
04:55 PM
@Zack Riesland ***Forum Tip: Try to avoid responding to an existing "Answer" by starting a new "Answer". There is no guaranteed order to answer, so conversation may get difficult to follow. Instead use the "Add comment" below an answer to respond. - That works as long as you have discipline amongst your various devs. With NiFi being unsecured there will be no controls preventing one user from modifying another users flows or deleting other users components. There also will not be an audit trail, as every change is logged as being done by anonymous. - Otherwise, a process group does provide user with what appears to be their own work space. - Keep in mind that it is still a single NiFi JVM, so every dataflow (no matter whom built it) is operating unders the same shared resource constraints (CPU, HEAP, etc..). - Thanks, Matt
... View more
05-23-2018
04:33 PM
@Zack Riesland Multiple users can access and make changes on the canvas at the same time. The only time that acton is blocked is if both users are trying to edit the exact same component at the same time. First user in such a scenario to hit "accept" on their changes wins. The other users change will be lost and they will be force to refresh and try again. - When NiFi is secured, you have the ability to setup very granular access controls. You can restrict different authenticated users for example to only have access to specific process groups. This would prevent users from editing components belonging to other users. - Thanks, Matt - If you found this answer addressed your question, please take moment to login and click "accept" below the answer.
... View more