@DivyaKaki    Cloudera's HDF offering provides Ambari managed NiFi, NiFi-registry, Zookeeper, NiFi CA, etc. services. https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/release-notes/content/hortonworks-dataflow-release-notes.html If you are looking for how to build your own custom Ambari management packs, I am not familiar with that process myself.   Thank you, Matt   ... View more

@Pr1    Not sure how often new files are written in to your HDFS directory, but you may want to look in to using the listHDFS and FetchHDFS processors.   The ListHDFS processor will list all FlowFiles based on processor configuration.  The processor then retains state based on timestamp of timestamp of the last execution.  Then it uses this timestamp so that only new Files since previous execution are listed during the next execution.  The ListHDFS processor only creates a 0 byte FlowFile for each File it lists.  The FlowFile includes attributes/metatdata about the HDFS file so that its content can later be fetched by the FetchHDFS processor.   The FlowFile produced by the ListHDFS processor then need to be routed to a FetchHDFS processor in order to retrieve the actual content from HDFS.   This model is designed so that the ListHDFS can be configured to run on the Primary Node only in a NiFi cluster and then you can distributed these 0 byte files across all nodes in your cluster before actually fetching the content (provides better performance by spreading the load across multiple servers)   The first run, you may get more than you wanted, but there are some configurations you can set to avoid or limit this in the listHDFS processor.  Consider using the "Maximum File Age" property.   Hope this helps, Matt ... View more

@saivenkatg55    You need to literally use  ./keystore.p12 in your command instead of just keystore.p12 curl --cert-type P12 --cert ./keystore.p12:password --cacert nifi-cert.pem -v https://w0lxqhdp04:9091/nifi-api/flow/search-results?q=     Hope this helps, Matt ... View more

@Gubbi    NiFi processor components offer one or more of the following scheduling strategies:   1. Timer Driven <-- When used, processor will request a thread every X amount of time as configured in the "Run Schedule". 2. Cron Driven <-- When used, processor will execute based on configured simple Quartz Cron in the "run schedule" configuration field.  http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html 3. Event Driven  <-- This is consider experimental and should not be used. Improvements to Timer Driven over the years has negated the usefulness of this strategy and it has been abandoned.  Only still exist to avoid breaking back wards compatibility for users who upgrade.  It will eventually be deprecated and removed.   None of these option will provide you with the ability to integrate with a calendar for specifically executing on US holidays.   That being said, everything you can do via the UI, you can also accomplish via external calls to the NiFi rest-api (this includes starting and stopping NiFi processors).  So technically you could create you own external scheduler that makes calls to NiFi's rest-api to start specific processors and then again to turn them off later. NiFi's rest-api documentation: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html   Hope this answers your question, Matt ... View more

@Sush217    First thing i noticed here is that it appears that your are trying to start all three NiFi nodes on a single Windows machine.  Why?  You just add overhead with three NiFi nodes on one server.  Typical NiFi cluster setups are one NiFi instance per server. The NiFi properties file would need to be unique per each NiFi instance.  They cannot share any binding ports in common since only one node is going to be able to bind to a given port number. This includes properties like: nifi.remote.input.socket.port= nifi.web.http.port= or nifi.web.https.port= (if secured) nifi.cluster.node.protocol.port= the configured NiFi hostname also needs to be resolvable.  You are using node1, node2, and node3.  Since all your NiFI nodes are running on the same Windows server, those would all need to be resolvable to the Windows server IP or you could use same hostname for all three instance since they are running on the same host. Now you are also trying to use the embedded zookeeper which means each instance us going to try to standup and start its own zookeeper node.  So just like NiFi, these zookeeper nodes can not share the same port numbers.  So each cannot be configured to use 2181, 2888, and 3888.  Plus, just like NiFi, the hostnames node1, node 2, and node 3 would all need to be resolvable in DNS or the windows local hosts file.  Also not ideal to have multiple zookeeper nodes running on the same server.   If you looking to just play around with NiFi, you should just install a standalone instance of NiFi, or setup a single node cluster.  You can always add additional nodes to your 1 node cluster later by setting up additional servers with NiFi. As you have it configured currently, my guess here is that if you parse through the nifi-app.log for each of your nodes, you are going to see errors related to resolving each your node hostnames to IPs and then later see address already in use errors. Hope this helps, Matt ... View more

@RAM17    The rest-api call you are making will not download the content of a FlowFile. A NiFi FlowFile consists of two parts: 1. FlowFile Attributes/metadata 2. FlowFile content You are looking to download the content of a FlowFile. NiFi does not track FlowFiles using the "filename".  NiFi only tracks FlowFiles by there assigned uuid.  So passing the filename " 9b3c4f34-ad98-4835-9412-94228db3adab.0.csv" is not going to result in content being returned. Using the NiFi rest-api to retrieve the content of a FlowFile from a connection queue is a multi step process. Step 1:  Make a queue listing request curl -X POST `http://localhost:8080/nifi-api/flowfile-queues/2dc8fdee-0170-1000-768a-ea14bc96a9cd/listing-requests` The json response will contain the listing request "id" which will be a uuid which will use in next rest-api call to get a listing of the FlowFIles in this queue. Step 2: Get list of FlowFiles created by above request. curl `http://localhost:8080/nifi-api/flowfile-queues/2dc8fdee-0170-1000-768a-ea14bc96a9cd/<listing request id>` Above will return json with the list of FlowFiles in the connection queue found under "flowFileSummaries" which will include following attributes for each FlowFile: --- "uri" rest-api uri for the flowfile --- "uuid" the UUID for the FlowFile (also found in above uri) --- "filename" filename attribute for flowfile  associated to above UUID. --- "position", "size", "queuedDuration", "lineageDuration", "penaltyExpiresIn", "clusterNodeId", "clusterNodeAddress", "penalized"   Step 3: Use the uri for the flowfile you want to retrieve the flowfile content from curl `http://localhost:8080/nifi-api/flowfile-queues/2dc8fdee-0170-1000-768a-ea14bc96a9cd/flowfiles/<flowfile uuid>/content` Performing above rest-api call without "/content" on the end will retrieve the FlowFile metadata instead of the content.  You also may want to redirect the response to above rest-api call to a file on local disk using the "filename" attribute returned during step 2.   Also keep in mind that you can only download the content of a FlowFile from a queue if that FlowFile is still in the connection queue.  If downstream processors are running and process that FlowFile from the connection queue, it will not be available any longer to download from that queue.   If you found this answer your query, please take a moment to accept the response.   Hope this helps, Matt   ... View more

@nhemamalini  Within NiFi you could construct a dataflow that uses the listHDFS processor to monitor your directory tree for creation of new files. This processor retains state so it will not list same files over and over each time it checks the HDFS directory tree.  An empty (no content) NiFi FlowFile will be created for each HDFS File listed which you can route to a FetchHDFS processor that would retrieve the content adding it to the FlowFile.   You can then do whatever notifications, transformations, etc via  the available processors in NiFi and use the publishKafika_<version> processor to send the file to your desired Kafka topic. Hope this helps, Matt ... View more

@JC_ROS  I struggle with how your NiFi communicates with a NiFi-Registry without clientAuth in your NiFi node's PrivateKeyEntry unless your NiFi-registry has not been secured.   The NiFi nodes must be authorized in your NiFi-Registry to both Read buckets and act as a proxy.  Authorizing a NiFi node requires that the node authenticate itself in the TLS handshake.  Unless of course you have authorized "anonymous". Thanks, Matt ... View more

@JC_ROS  With a secured NiFi all actions are authenticated and authorized.  Thus mutual TLS authentication is required for NiFi things like: 1. Node to Node communications (Older versions of NiFi had ability to set NeedClientAuth=false in the nifi.properties to make node to node comms 1-way TLS.  Newer NiFi releases removed this property resulting in mutual TLS being required here. 2. NiFi Remote Process Groups (RPG) - The S2S protocol used by the RPG required mutual TLS.  The NiFI instance running the RPG is acting as a client and must present a clientAuth certificate. 3. NiFi load balanced connections The above all utilize the NiFi node keystore and truststore.  Hope this helps, Matt ... View more