@NickH    If the listSFTP processor fails during listing, no FlowFiles should have been output and the state should not have been updated.  Are you seeing failure during the listSFTP processor execution?   If you are seeing FlowFiles getting routed to one of the failure relationships from the FetchSFTP processor, you can always loop that connection back to the same FetchSFTP processor so another attempt is made to fetch the content for that FlowFile. There currently does not exist a may to clear just a single cached entry from the DistributedMapCacheServer controller service.  I encourage you to open an Apache NiFi Jira for a new processor that can remove cache entries. https://issues.apache.org/jira   You could try looking at this example for removing a cache entry via a script: https://gist.github.com/ijokarumawak/14d560fec5a052b3a157b38a11955772     Hope this helps, Matt ... View more

@Alexandros    You might want to look at using the Site-To-Site Bulletin Reporting Task.  Anytime a component produces an ERROR or WARN log output (For example when a FlowFile is routed to a processor's failure relationship), a bulletin is produced on the component.  The " SiteToSiteBulletinReportingTask" would get that bulletin and send it via the S2S protocol to another NiFi instance/cluster where you would build a notification dataflow using for example the putEmail processor.  Keep in mind that having a different NiFi is not required, you can have the "SiteToSiteBulletinReportingTask" just sent the bulletins to a flow on the same NiFi that your are monitoring.   This scales well as new components are added to your canvas and does not require separate monitoring on every flow.   Hope this helps, Matt ... View more

@asfou    NiFi does not contain any processors that support Hive version 2.x.  The latest versions of Apache NiFi offer Hive 1.x and Hive 3.x client based processor components.  To support Hive 2.x version, you may need to build your own custom processors built using the Hive 2.x client. Matt ... View more

@saivenkatg55    By "hung", you mean the processor shows active thread(s) in the upper right corner continuously and never produce any ERROR or WARN log output?  Active threads show as a small number in parenthesis (1).  If that is the case, you would need to get a series of thread dumps from the NiFi node where the active thread is executing.  5 dumps taken with 5 minutes between each thread dump is usually pretty good. Then you will want to inspect those thread dumps for the active thread associated to your hung processor type.  Look to see if the thread changes between any of the thread dumps.  If the thread output in the dumps is changing then the thread is not hung, but rather just taking long time to execute.  If the thread is identical across all thread dumps, you'll want to look at the thread to see fi it identifies what the thread is waiting on.   Hope this helps, Matt ... View more

@saivenkatg55  My first suggestion is that you always make sure you have configured the "validation query" property when using the DBCPConnectionPool controller service. When you enable the DBCPConnectionPool it does not immediately establish the configured number of connections for the pool.  Upon receiving the first request from a processor for a connection from the DBCPConnectionPool, the entire pool will be created (by default configuration pool would consist of 8 connection). Subsequent requests by processor to for a connection results in one of the previously established connections being handed off to the requesting processor.  If and only if the validation query is configured, the connection is first validated to make sure it is still good before being passed to requesting processor.  So often times we see this issue when validation query is not set and the connections are no longer any good.  Connections can go bad for many reasons (network issues, idle to long, etc...) You should keep your validation query very simple so it can execute very fast.  "select 1" is often used to validate a connection. Also make sure you are using the latest version of NiFi.  There have been numerous bug fixes around the processors and DBCPConnectionPool between Apache 1.0 and 1.11. Hope this helps, Matt ... View more

@DivyaKaki    Cloudera's HDF offering provides Ambari managed NiFi, NiFi-registry, Zookeeper, NiFi CA, etc. services. https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/release-notes/content/hortonworks-dataflow-release-notes.html If you are looking for how to build your own custom Ambari management packs, I am not familiar with that process myself.   Thank you, Matt   ... View more

@Pr1    Not sure how often new files are written in to your HDFS directory, but you may want to look in to using the listHDFS and FetchHDFS processors.   The ListHDFS processor will list all FlowFiles based on processor configuration.  The processor then retains state based on timestamp of timestamp of the last execution.  Then it uses this timestamp so that only new Files since previous execution are listed during the next execution.  The ListHDFS processor only creates a 0 byte FlowFile for each File it lists.  The FlowFile includes attributes/metatdata about the HDFS file so that its content can later be fetched by the FetchHDFS processor.   The FlowFile produced by the ListHDFS processor then need to be routed to a FetchHDFS processor in order to retrieve the actual content from HDFS.   This model is designed so that the ListHDFS can be configured to run on the Primary Node only in a NiFi cluster and then you can distributed these 0 byte files across all nodes in your cluster before actually fetching the content (provides better performance by spreading the load across multiple servers)   The first run, you may get more than you wanted, but there are some configurations you can set to avoid or limit this in the listHDFS processor.  Consider using the "Maximum File Age" property.   Hope this helps, Matt ... View more

@saivenkatg55    You need to literally use  ./keystore.p12 in your command instead of just keystore.p12 curl --cert-type P12 --cert ./keystore.p12:password --cacert nifi-cert.pem -v https://w0lxqhdp04:9091/nifi-api/flow/search-results?q=     Hope this helps, Matt ... View more

@Gubbi    NiFi processor components offer one or more of the following scheduling strategies:   1. Timer Driven <-- When used, processor will request a thread every X amount of time as configured in the "Run Schedule". 2. Cron Driven <-- When used, processor will execute based on configured simple Quartz Cron in the "run schedule" configuration field.  http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html 3. Event Driven  <-- This is consider experimental and should not be used. Improvements to Timer Driven over the years has negated the usefulness of this strategy and it has been abandoned.  Only still exist to avoid breaking back wards compatibility for users who upgrade.  It will eventually be deprecated and removed.   None of these option will provide you with the ability to integrate with a calendar for specifically executing on US holidays.   That being said, everything you can do via the UI, you can also accomplish via external calls to the NiFi rest-api (this includes starting and stopping NiFi processors).  So technically you could create you own external scheduler that makes calls to NiFi's rest-api to start specific processors and then again to turn them off later. NiFi's rest-api documentation: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html   Hope this answers your question, Matt ... View more

@Sush217    First thing i noticed here is that it appears that your are trying to start all three NiFi nodes on a single Windows machine.  Why?  You just add overhead with three NiFi nodes on one server.  Typical NiFi cluster setups are one NiFi instance per server. The NiFi properties file would need to be unique per each NiFi instance.  They cannot share any binding ports in common since only one node is going to be able to bind to a given port number. This includes properties like: nifi.remote.input.socket.port= nifi.web.http.port= or nifi.web.https.port= (if secured) nifi.cluster.node.protocol.port= the configured NiFi hostname also needs to be resolvable.  You are using node1, node2, and node3.  Since all your NiFI nodes are running on the same Windows server, those would all need to be resolvable to the Windows server IP or you could use same hostname for all three instance since they are running on the same host. Now you are also trying to use the embedded zookeeper which means each instance us going to try to standup and start its own zookeeper node.  So just like NiFi, these zookeeper nodes can not share the same port numbers.  So each cannot be configured to use 2181, 2888, and 3888.  Plus, just like NiFi, the hostnames node1, node 2, and node 3 would all need to be resolvable in DNS or the windows local hosts file.  Also not ideal to have multiple zookeeper nodes running on the same server.   If you looking to just play around with NiFi, you should just install a standalone instance of NiFi, or setup a single node cluster.  You can always add additional nodes to your 1 node cluster later by setting up additional servers with NiFi. As you have it configured currently, my guess here is that if you parse through the nifi-app.log for each of your nodes, you are going to see errors related to resolving each your node hostnames to IPs and then later see address already in use errors. Hope this helps, Matt ... View more