@Abhinav Joshi I suspect that you still have multiple versions of the updateAttribute loaded in your new NiFi 1.9.0 install. My guess here is that the updateAttribute-1.4.nar was added to your NIFi 1.8 and NiFi 1.9 as an additional nar manually. Check your nifi.properties file in NiFi 1.9.0 for the every occurrence of "nifi.nar.library.directory". You may find 1 or more. These will be the locations from which your NiFi is loading its nars. Then look in these directories for "nifi-update-attribute-nar" to see if you find multiple versions in use. I am guessing that in your NiFi 1.9.0 you will find a 1.9.0 version of this nar and a 1.4.0 version. - So what happens when the flow.xml.gz file you are using from NiFi 1.8.0 is loaded, you have multiple versions of the updateAttribute processor (1.4 and 1.8). The 1.4 versions load just fine because the 1.4.0 nar still exists; however, there is no 1.8.0 version of the nar. Normally NiFi would auto select the available option, but since there are two options available (1.4 and 1.9 now), NiFi cannot auto select because two options exist and ghost implementation of the 1.8 versions are created requiring the user to manually replace them with the desired version. - So you have two options: 1. Remove the 1.4 nar so only 1.9 version of the nar exists 2. Manually edit the flow.xml.gz file replacing the version number on the 1.8 updateAttribute processors with the new 1.9 version number. - If you did not find multiple nars, then perhaps you are new 1.9 install is using the same old work directory as the original 1.8. NiFi unpacks the nars in the work directory on startup. IN this scenario simply delete everything in the work directory before starting NiFi so it is rebuild form the nars/jars found in the above mention library directories. - Thanks, Matt ... View more

@Samar Aarkotti *** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion. It always best to leave your processor at default value for concurrent task unless there is a specific need to increment. Here is an article on this topic: https://community.hortonworks.com/articles/221808/understanding-nifi-max-thread-pools-and-processor.html and another on "Run Duration": https://community.hortonworks.com/articles/221807/understanding-nifi-processors-run-duration-functio.html ... View more

@Kevin Lahey 1. Each NiFi node in a cluster runs its own copy of the flow.xml and processes its own set of FlowFiles. Node are unaware of what FlowFiles exist on other nodes in the cluster. 2. In much older versions of NiFi (Apache 0.x versions), NiFi did not have any High availability at the control level within a cluster. There existed a dedicated NiFi instance known as the NiFi Cluster Manager (NCM). This was the only instance in the NiFi cluster that could be accessed. All the nodes connected to this NCM. If NCM went down the entire NiFi cluster was not reachable. As of Apache NiFi 1.x+ the NCM no longer exists and the cluster relies on Zookeeper to elect one of the cluster nodes to handle role of Cluster Coordinator and Primary node. If the currently elected node(s) for these roles goes down, a new load is elected to these roles. In this way HA at the control level was provided. When you create any component (processor, controller service, reporting task, etc...), those components are replicated to all nodes in the cluster. So yes, the DistributedMapCacheServer controller service would be running on all nodes. If you then configured the DistributedMapCacheClient to use "localhost", then each node would be reading and writing to different cache servers. The DistributedMapCacheClient should be configured to point at a specific node rather than localhost. As you can see you have no HA in this type of setup since you are dependent on that one node hosting the cache server you are using to always be up. Instead you shoudl be using one of the external cache options like HBase in order to have HA. 3. As explained above, there is not such thing as a NCM as of Apache NiFi 1.x+ 4. Every component you add to the NiFi canvas is running within a single JVM on each NiFi node. So you cannot configure multiple components that bind to the same configured port anywhere. The first component will bind to port and when the other components are started they will throw an exception about port already in use. You can have as many clients (DistributedMapCache Client) as you like, since they act as a client and do not bind to a port. Only the server binds to the port so it can listen for client requests. Hope this helps ... View more

@Abhinav Joshi *** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion. I would suggest searching the nifi/work directory for multiple versions of the update-attribute nar bundle. You may have multiple nars of different versions installed. The flow.xml.gz file does contain the specific processor version for each component. When starting NiFi 1.9 using the flow.xml.gz from another NiFi version, the component versions will automatically be updated to the new version only if a single option exists. If you have an updateAttribute-1.8.<custom> and an updateAttribute-1.9.0 version available and the flow.xml.gz has an updateAttribute-1.8.0 then it will not auto-update because there are two options and it does not know which should be used. - My guess here is that your NiFi 1.8.0 contained both the standard 1.8.0 version of the the updateAttribute processor and a custom version of the updateAttribute processor. Then your flow contained updateAttribute components of each, Then you upgraded to NiFi 1.9.0 which replaced the stock updateAttribute with 1.9.0 and the custom version of Update Attribute processor was also carried over to your NiFi 1.9.0 install. - Thanks, Matt ... View more

@Kevin Lahey I completely agree with @Shu. I sounds like you have ListS3 processor executing on all 4 nodes in a NiFi cluster. This results in each NiDi node listing the same filename. This means that each node is then trying to lookup that filename in the distributed cache used by the detectDuplicate processor. This results in a bit of a race condition between you nodes where one or more nodes fails to find entry in cache before 1 of the nodes adds this new filename to that cache. - You flow should be running the ListS3 processor with it success relationship feeding a FetchS3 processor. That connection between those two processors should be configured to load balance the listed files across all nodes in cluster. - Thanks, Matt ... View more

@Abhinav Joshi A processor will appear with a dashed line around perimeter of the processor box for a couple reasons 1. It is a ghost implementation --> when loading the flow.xml.gz a processor is encountered that uses a custom nar which does not exits in the current NiFi installation. 2. The user logged into the canvas does not have required permissions to view the component. - Based on your description, it does not sound like scenario 2. The question is why did NiFi 1.9 not find a processor in NiFi 1.9 that used same processor class. Since you upgraded from NiFi 1.8, the update Attribute processor should have referenced the org.apache.nifi.processors.attributes.UpdateAttribute class for version 1.8.x. During an upgrade of NiFi the same identical class would have been found except for a newer version. In this case NiFi would have automatically switched to using the new version. - So this raises two questions: 1. Did you customized version of the updateAttribute processor running in NiFi 1.8? 2. Do you have multiple copies of the same processor class but with different versions loaded in your NiFi 1.9? - I would suggest inspecting the nifi-app.log from the time of startup immediate following the upgrade. If the UpdateAttribute processors was replaced by a "ghost" processor you would see that logged in the nifi-app.log. This log should help to see why it chose to load a ghost processor. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link. ... View more