@Jobin George When you add new components (Process groups or processors), they inherit the policies from the parent component by default. This means the your process group (Group1) has inherited some policies maybe from its parent process group and your processor (getSFTP) has inherited policies from the process group it is inside. My guess is that those inherited policies are allowing user "john" view and "modify" to process group 'Group1'. When you select a component (process group or processor) and click on the key icon to modify/set its policies, you may notice the following in the "Access Policies" UI that is displayed: This line is telling you that the policies you are currently looking at are coming from a parent process group. If you modify any of these policies, what you are really doing is modifying the policies on that parent process group rather then on the actual selected component. In order to set specific policies for the select component, you must fist click on "Override". You will then see the above effective policy line go away and the specific policy you are currently looking at will be cleared of all entries. Now you can add specific users for this policy that are applied to only tis component. If the component is a process group, any processor or additional process group within will inherit this new policy. Keep in mind that every policy inherits from its parent by default, so clicking on "Override" only create a new policy accesses for that one policy. You will need to select each available policy for a component and click "Override" for each one where you want to set component specific policy accesses. Thanks, Matt ... View more

@Gerd Koenig After a closer look at the jaas file you posted above, I believe you issue is because of a missing " in the following line: principal="nifi@REALM; This line should actually be: principal="nifi@REALM"; Try making the above change and restarting your NiFi. Thanks, Matt ... View more

@kishore sanchina The simplest answer to your question is to use the ListFile processor to produce a list of the files from your local filesystem, feed that to a fetchFile processor that will pickup the content and then pass them to a PutHDFS processor to send them to your HDFS. The listFile processor will maintain state based on lastModified time on the files to ensure the files are not listed more then once. If you right click on either of these NiFi processors you can select "usage" from the displayed context menu to get more details on the configuration of each of these. Thanks, Matt ... View more

@Hans Feldmann The individual processors allow for concurrent task changes. By default they all have one concurrent task. For each additional concurrent task, you are giving that processor the opportunity to request an additional thread from the NiFi controller to do work in parallel. (Think of it as two copies of the same processor doing working different files or batches of files). If there isn't sufficient files in the incoming queue then any additional concurrent tasks are not utilized. The flip side is if you allocate two many concurrent tasks to a single processor, that processor may itself end up using two many threads from the NiFi controller's resource pull resulting in a thread starvation to other processors. So star with the default and setup by one increment at a time in place of backlog in your flow. The NiFi controller also has a setting that limits the maximum number of threads it can use from the underlying hardware. This is the other thing Andrew was mentioning. A restart of NiFi is NOT needed when you make changes to these values. The defaults are low (10 timer driven and 5 event driven). I would set the timer driven to no more then double the number of cores your hardware has. Thanks, Matt ... View more

Yes you can use that state directory and just create the zookeeper sub directory in which you will have the myid file. I do recommend that your state directory is instead created somewhere outside of the base NiFi install path. This can aid in simplifying future upgrades of NiFi. Since newer version will still want to reference the existing cluster wide state created in your existing NiFi version. If you do choose to move it form default, update the zookeeper properties file and create the new path. ... View more