@Ahmad Debbas The GetHDFS processor is deprecated in favor of using ListHDFs and FetchHDFS processors. The GetHDFS processor does not retain state and therefore will start over from the beginning as you noted when an error occurs. The ListHDFS processor does maintain state, so even through NiFi restarts or processor restarts, the listing picks up where it left off. The zero byte FlowFiles produced are then passed to a FetchHDFS that actually retrieves the content and inserts it into the existing FlowFile. Another advantage to the list/fetch design model is the ability to distribute those listed zero byte files across a Nifi cluster before fetching the content. This improves performance by reducing resource strain caused by GetHDFS on a single NiFi node. Thanks, Matt ... View more

@Pushkara Ravindra The intent of the Site-To-Site (S2S) protocol is to allow the exchange of NiFi FlowFiles between NiFi instances. A NiFi FlowFile consists of two parts: 1. FlowFile content <-- Original content in whatever format (NiFi is data agnostic and has no data format dependency) 2. FlowFile Attributes <-- Collection of key/value pairs (some are NiFi assigned by default while others are add via processors) Sending FlowFiles between NiFi instances allows the originating NiFi to share the attributes it knows about a FlowFiles content with the target NiFi instance. The FlowFile Attributes are loaded in to the FlowFile repo of the target NiFi automatically. In addition to the above, the S2S allows for the automatically smart load-balancing of FlowFiles to a target NiFi cluster. S2S allows for the auto-scaling up or down of the target Nifi cluster without the client needing to change anything. How it all works: The source/client NiFi instance/cluster will add a Remote Process Group (RPG) to their canvas and configure it to point at the URL of any target/destination NiFi instance or cluster node. The communication at this point is over HTTP protocol. Once a connection is established the destination NiFi sends S2S details back to the source NIFi (Includes URLs of nodes if destination is cluster and the current load of each node.) The RPG will continuously update this information and store a local copy of this information in the event it cannot get an update at any time. Input and output ports are used to send or receive FlowFiles from the parent process group of where they were added. So when input or output ports are added to the root canvas level of any dataflow they become "remote" input and output ports capable of sending or receiving data from another NiFi. Whether you set the S2S protocol to HTTP or RAW the above is true. What is different is what happens next (Actual FlowFile transfer). When using the RAW format (Socket based transfer), the "nifi.remote.input.host" and "nifi.remote.input.socket.port" configured values from each of the target NiFi instances are used by the NiFi client as the destination for sending FlowFiles. When using the HTTP format, the "nifi.remote.input.host" and the "nifi.web.http.port" or "nifi.web.https.port" configured values from each of the target NiFi instances are used by the NiFi client as the destination for sending FlowFiles. Advantage of RAW format is that their is a dedicated port for all S2S transfers, so under high load it affect on the NiFi HTTP interface is minimal. Advantage of HTTP, you do not need to open an additional S2S port since teh same HTTP/HTTPS port is used to transfer FlowFile. Thanks, Matt ... View more

@Bram Klinkenberg The users.xml and authorizations.xml files are generated for you the first time NiFi is started after being secured. Initially they are populated using the configuration from the authorizers.xml file. In that file you specified an "Initial Admin Identity" (assuming you used CN=admin). As a result a user (CN=admin) was added to the users.xml file and the relevant "admin" related access policies were assigned to that user in the authorizations.xml file. At this point your user (CN=admin) should be able to access the NiFi UI. The admin will use the NIFi UI to add additional users and authorize them for various access policies: Users are managed and Global Policies are applied as follows: Adding "Users" within NiFi has nothing to do with user authentication. The users you add here are for authorization to NiFi resources only. User Authentication must occur first and can be accomplished using User issued certs load in browser, Kerberos, or LDAP. Global access policies include the following: Component (Processors, process groups, and other things on canvas) level access policies are assigned to users as follows: Component level access policies include: Some Component level access policies are on available to specific components. If the currently selected component does not support the policy it will be greyed out in the list. More detail on teh various access policies can be found in teh admin guide: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#config-users-access-policies Thank you, Matt ... View more

@Nikhil Chaudhary The encryption of values in the NiFi variable registry is not there yet. It is a future goal in Apache NiFi. There is an existing Apache Jira that covers the adding of this capability: https://issues.apache.org/jira/browse/NIFI-2653 Thanks, Matt ... View more