@uttam kumar Using the NiFi REST-API, i was able to upload, delete, and upload again the same template using the following commands To upload: curl -X POST -v -F template=@"/<path to template>/template.xml" http://<host>:<port>/nifi-api/process-groups/<UUID of process group>/templates/upload The UUID of my template was in the response to the above command. To delete: curl -X DELETE -v http://<host>:<port>/nifi-api/templates/<UUID of template> I then repeated the above to commands successfully several times. Thanks, Matt ... View more

@nyakkanti Only processor properties that support the NiFi expression language can be configured to use FlowFile attributes. Some properties (even if the processor property says supports expression language: false, you may be able to use a simple ${<FlowFile-attribute-name>} to return a value from a attribute key/value pair. But this is never true for sensitive properties such as password properties. Password properties are especially difficult since those values are encrypted upon apply. So if you entered ${<FlowFile-attribute-name>} , that is what gets encrypted. The processor then decrypts that when making the connection. Nowhere in that process does it ever retrieve a value from the FlowFile. Thanks, Matt ... View more

@ismail patel When the Remote Process Group was originally written, the list and fetch type processors did not exist and NiFi was primarily being used to process large files. Based off that, the design used worked pretty well for load-balancing. Here is how it works: 1. RPG connects to target NiFi cluster. 2. Target NiFi returns a list of available peer nodes along with their current load. 3. RPG creates a distribution allocation. (You will see in logs that distribution in the from of percentages. Node 1 50%, Node2 50%). 4. RPG connects to node 1 and sends data from incoming queue for 5 seconds. then connects to node 2 and sends for 5 seconds. Lets assume a distribution las follows: Target is 4 node cluster and base on load distribution, the following distribution was calculated: Node 1 -- 10% Node 2 -- 20% Node 3 -- 20% Node 4 -- 50% As a result the RPG would connect in this pattern to send the data: Node1, Node2, node3, node4, node2, node3, node4, node4, node4, node4. While with a high volume dataflow and a dataflow dealing with larger files this balances out nicely over time. A low volume flow or a flow dealing with very small files, it does not work so well. In your case, you have a bunch of small files (all 0 byte), so in 5 seconds every one of them is send to node 1. There are improvements coming to the RPG to set batch sizes and improve how the distribution of data occurs. Thank you, Matt ... View more

@Bharadwaj Bhimavarapu Whether you are using the ConsumeKafka or PublishKafka processors, if Kafka is kerberized you will need to setup a JAAS file in your NiFi which provides the keytab and principal used to establish that secured connection. By default the /etc/krb5.conf will be used, but you can also tell NiFi to use a different krb5.conf file via a property in the nifi.properties (nifi.kerberos.krb5.file=). You will need to create a JAAS file (example: kafka-jaas.conf) that contains the following (Update to use appropriate keytab and principal for your user): KafkaClient{   com.sun.security.auth.module.Krb5LoginModule required   useKeyTab=true   storeKey=true   keyTab="nifi.keytab"   serviceName="kafka"   principal="nifi@DOMAIN"; }; Add the following line to NiFi's bootstrap.conf file (make sure arg number 20 is not already being used, if so change to an unused number): java.arg.20=-Djava.security.auth.login.config=/<path-to>/kafka-jaas.conf Update the following configuration properties in your ConsumeKafka processor: SecurityProtocol= SASL_PLAINTEXT ServiceName= kafka Basically you are setting up the Kafka client kerberos environment for your NiFi JVM. If this is a NiFi cluster, you will need to to the above on every node. You will need to restart NiFi for these changes to take affect. Thanks, Matt ... View more

@Jatin Kheradiya There are a few things that do not look right in your nifi.properties configuration above: On every node the following properties should be configured with the FQDN of the node: 1. nifi.remote.input.host= 2. nifi.cluster.node.address= 3. nifi.web.http.host= or nifi.web.https.host= I noticed you are configuring nifi.remote.input.host= with the IP of a different node. It is not clear from the above if you set a value for nifi.web.http.host= or nifi.web.https.host=. If you did not, Java may be resolving your hostname to localhost. This can be problematic for cluster communications. Since node may end up trying to talk to themselves rather then actually talking to the other nodes. Also make sure that the following ports are open in any firewalls between your nodes: 1. nifi.remote.input.socket.port= 2. nifi.cluster.node.protocol.port= 3. nifi.web.http.port=8080 or nifi.web.https.port= Also make sure all three of your nodes can talk to zookeeper on port 2181. Thanks, Matt ... View more