Member since
07-30-2019
3427
Posts
1632
Kudos Received
1011
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 85 | 01-27-2026 12:46 PM | |
| 491 | 01-13-2026 11:14 AM | |
| 1028 | 01-09-2026 06:58 AM | |
| 916 | 12-17-2025 05:55 AM | |
| 977 | 12-15-2025 01:29 PM |
10-13-2017
04:01 PM
@Hadoop User If merging FlowFiles and adding more concurrent tasks to your putHDFS processor help with your performance issue here, please take a moment to click "accept" on the above answer to close out this thread. Thank you, Matt
... View more
10-13-2017
12:49 PM
@Anishkumar Valsalam If the answer provided solved your issue, please take a moment to click "Accept" below the answer to close out this thread. This will assist other who are searching for a solution to the same problem. Thank you, Matt
... View more
10-12-2017
09:22 PM
3 Kudos
@Anishkumar Valsalam You have encountered the following: https://issues.apache.org/jira/browse/NIFI-4363 You can get around this issue by editing the following script on every NiFi node in your cluster: /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/files/nifi-toolkit-1.2.0.3.0.1.0-43/bin/encrypt-config.sh Near the bottom of the script you will find this line: "${JAVA}" -cp "${CLASSPATH}" -Xms128m -Xmx256m org.apache.nifi.properties.ConfigEncryptionTool "$@" Edit it so that it now says: "${JAVA}" -cp "${CLASSPATH}" -Xms512m -Xmx1024m org.apache.nifi.properties.ConfigEncryptionTool "$@" Then go back to Ambari and start your NiFi service. Thank you, Matt
... View more
10-12-2017
02:29 PM
@Dhamotharan P I highly recommend staring a new HCC question for your particular use case so it gets better visibility. You van always reference this HCC in your new question. The bulletin and any stack trace from the nifi-app.log when you take your end-point down would also be very helpful. Perhaps it is doing a session rollback instead which leaves the file on the inbound connection rather the routing to failure. Thank you, Matt
... View more
10-03-2017
05:22 PM
@dhieru singh Only processor properties that explicitly state that they support NiFi Expression Language will allow you to set variables. The "Port" property requires that a hard coded integer value is supplied. There is work being done to expand the capability of the NiFi variable registry. That work will expand what processor properties support EL. I recommend opening an Apache Jira requesting that the "Port" properties add support for NiFi expression language. Until then, there is no workaround. The value must be supplied directly in the processor. Thanks, Matt
... View more
10-03-2017
12:57 PM
@Andre Labbe Another tip: use the @<username> in your responses/comments so that the user is notified when you add a response/comment.
... View more
10-03-2017
12:56 PM
@Andre Labbe Tip: try to avoid creating new answers for every correspondence. Add comments to existing answers to continue that discussion thread. 1. It is possible to have a one node cluster, but not very efficient. By setting it up as a cluster you are adding the overhead of zookeeper that is not needed with a standalone Nifi install. With regards to your "An unexpected error has occurred", this has nothing to do with there only being one node. Something else is going on in that would need to be investigated via the nifi logs (nifi-app.log, nifi-user.log, and/or nifi-bootstrap.log). 2. ExecuteScript->ExecuteStreamCommand->ExecuteScript-> ExecuteScript->InvokeHTTP. (That is a lot of custom code in a dataflow.) Do you really need so many script/command based processors to satisfy your use case? You also must remember that each node runs its own copy of the dataflow and with the exception of the cluster coordinator, has no notion of the existence of other nodes. A node simply run the dataflow independently and works with it sown set of files and NiFi repos. So make sure that your scripts exist on every node in your cluster in the same directory locations with proper permissions. There is nothing fancy about zookeeper needed by NiFi. Zookeeper is just another Apache project and has its own documentation ( https://zookeeper.apache.org/ ). You do not need a zookeeper node for every NiFi node. A 3 node ZK is perfect for most NiFi clusters. If you found this answer addressed teh question asked in this forum, please click "accept" . Thanks, Matt
... View more
10-02-2017
05:43 PM
@Andre Labbe 1. NiFi is designed to prevent changes while a node is disconnected. Each node runs its own copy of the flow.xml.gz. When a node is disconnected, the cluster coordinator has not means to determine the state of components on that disconnected node. If changes were allowed to be made, that node would be unable to automatically rejoin the cluster without manual intervention to bring them back in sync. To regain control of the canvas you can access the cluster UI and drop the disconnected node from the cluster. Keep in mind that if you want to later rejoin that node, you will need to make sure the flow.xml.gz, users.xml (if used when NiFi is secured), and authorizations.xml (if used when NiFi is secured) are all in-sync with what is currently be used in cluster (you can copy these from other active cluster node). *** Be mindful that if a change you made removed a connection that contained data on your disconnected node, that data will be lost on startup once you replace the flow.xml.gz. If Nifi cannot find the connection to place the queued data it is lost. 2. If you set a Processor component to execute on "Primary node", it dose not run. What component and how do you know it did not run? If have not see this happen before. Processors that use no cluster friendly protocols should be run on Primary Node only to prevent data duplication as you noted above. If you are consuming a lot of data using one of these protocols, ti is suggested you use the List/Fetch (example: listSFTP/FetchSFTP) processors along with NiFi's Site-To-Site (S2S) capability to redistribute the listed FlowFiles to all nodes in your cluster before the Fetch. Also note that you need an odd number of Zookeeper nodes in order to have Quorum. (3 minimum). Using the embedded zookeeper means you lose that ZK node anytime you restart that NiFi instance. I don't recommend using embedded ZK in production. Thanks, Matt
... View more
09-27-2017
06:47 PM
@pawan soni
Did you resolve your invalidate state by starting your "From File" input port? Your screenshot shows the RPG as "Enable transmission" and the input port as "stopped". Thanks, Matt
... View more
09-27-2017
06:26 PM
@Obaid Salikeen There is no direct correlation between CPU and heap memory usage. Heap usage is more processor and flow implementation specific. Processors that do things like splitting or merging of FlowFiles can end up using more heap. FlowFile Attributes live in heap memory. NiFi does swap FlowFile attribute to disk per connection based on FlowFile queue count. Default of 20,000 will trigger swapping to start on a connection. But there is no sap threshold based on FlowFile attribute map size. If a user creates large attribute values to FlowFile Attributes, that FlowFile heap usage is going to be higher. You see this isn scenarios where large parts of the FlowFile content is extracted to a FlowFile attribute. So when it comes to heap/memory usage, it comes down to flow design more then any correlation to the number of CPUs. Thanks, Matt
... View more