@Toky Raobelina I am really not sure what version of NiFi you are running. The Authentication strategy "LDAPS" is different then "START_TLS". LDAPS support in NiFi's login identifier is a fairly new addition. Consult your NiFi's imbedded admin guide documentation to verify that LDAPS is an option. If so you will want to first change your login-idnetity-providers.xml fiel configuration to use LDAPS instead of START_TLS. Next you will want to confirm your ldap servers URL. Typically ldaps urls start with ldaps:// instead of just ldap://. You also mentioned that you add the ldaps servers public key to NiFi's keystore. NiFI use 2-way TLS authentication. You should have added the ldaps servers public key as a trustedCertEntry in NIFi's truststore instead of the keystore. Also you want to make sure you have added NiFi's nodes public key as a trustedCertEntry on the ldaps server as well. If all you NIfi certs where signed by a CA, you just need to add the public key for your CA as a trustedCertEntry instead. Thank you, Matt ... View more

@Muhammad Umar You have to remember that NiFi is being run by some user and that user is running all these processor components. NiFi's base directory is not going to be the same directory as where you point to the script. Try passing the absolute directory path to the script inside of NiFi instead of just "nifi-srcfiles/*" and "BackupFiles/". Thank you, Matt ... View more

@bhumi limbu That is a very useful ERROR log message. You appear to be running in to teh following reported issue: https://issues.apache.org/jira/browse/NIFI-3096 Thanks, Matt ... View more

@Gaurav Jain Was I able to successful answer your question? If so please mark the answer as accepted. Thank you, Matt ... View more

@Muhammad Umar When NiFi starts and has not been configured with a specific hostname or IP in the (nifi.web.http.host=) in the nifi.properties file, it looks to bind to the IP address registered to every NIC card present on the host system. If you try to specify a hostname or IP that does not resolve or match the IP registered to any of your NIC cards, NiFi will fail to start. NiFi can not bind to a port that belongs to an IP it does not own. You can run the "ifconfig" command on the host running NiFi to see all NICs and the IP registered to them. You should see the 172.17.x.x address and not the 192.168.x.x address shown. It definitely sounds like there is some network address translation going on here. The fact that you can reach NiFi over http://192.68.x.x:8078// confirms this. It is simply routing all traffic from the 192.169.x.x address to the internal 172.17.x.x address. We confirmed already your browser cannot resolve a path directly to 172.17.x.x, because if you could, NiFi's UI would have opened. NiFi is in fact bound to 172.17.x.x and not 192.168.x.x. NiFi cannot control how traffic is being routed to this endpoint by the network. Thanks, Matt ... View more

@Gaurav Jain If the NiFi node suddenly goes down, how is it going to notify the other nodes of this? If the node goes down, the result of the job is neither a failure or success as NiFi defines it. The FlowFile that triggers your SparkJobExecutor should remain tied to the incoming connections until the job successfully completes or reports a failure. At that time the FlowFile would be moved to one of the corresponding relationship. If the NIFi node goes down, when it comes back up FlowFiles are restored to the last known connection. This means this FlowFile will trigger your SparkJob executor to run again. Are you looking for a way to notify another node that the last spark job did not complete only or are you also looking for a way for that other node to then run the job? That becomes even more difficult since you must also tell the node that went down not to run again next time it starts back up. As far as the notification goes, you might be able to build a flow using the new wait and notify processor just released in Apache NiFi 1.2.0. You could send a copy of your FlowFile to another one of your nodes before executing the Spark job and then send another FlowFile after job completes. On the other node it should receive the FlowFile and send it to a wait processor. The wait processor can be configured for a time limit. Should that time expire the FlowFile gets routed to expired relationship which can use run the job again on that node or simply send out a email alert. If the job completes before expiration time occurs, a FlowFile could be send to same node to notify successful completion which will cause wait processor to rout FlowFile to success relationship which you may choose to just terminate. Here is the doc for these new processors: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.Wait/index.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.Notify/index.html Bottom line, NiFi has no behind the scenes monitoring capability to accomplish what you are trying to do here, so a programmatic dataflow design must be used to meet this need. Now if you are talking about a node simply becoming disconnected from the cluster, that is a different story. Just because a node disconnects does not mean it shuts down or stops running its dataflows. It will continue to run as normal and constantly attempt to reconnect. Thanks, Matt ... View more

Both your NiFi clusters can use teh same zookeeper, but you need to make sure each cluster is configured to use a different ZK root node. The root node is set in the nifi.properties file and the state-management.xml file. ... View more