@Sanaz Janbakhsh The policies you have identified above /flow (grants users the ability to view the UI), /proxy (Allows NiFi nodes and proxy servers to proxy requests for users to other NiFi nodes), default "all-nifi-resources" assigns "*" which grants user here access to every policy. The component level granular policies are based on the components assigned uuid. For connections, the policies are enforced based upon the processor component the connection originates from. for example: /remote-process-groups/<remote process group uuid> /data/remote-process-groups/<remote process group uuid> /process-groups/<process group uuid> /data/process-groups/<process group uuid> /processors/<processor uuid> /data/processors/<processor uuid> There will be a unique policy for each and every component based on the specific components assigned uuid available. Component level authorizations are inherited from the parent process group when no specific processor, remote-process-group, or sub process group component level policy is set. So for a user to be able to view the FlowFiles in a connection (list queue), they must be granted "read" for the component (/data/processors/<processor uuid>) from which that connection originated. Access can be granted via inheritance from a parent process group instead by granting the user "read" to a parent process group (/data/process-groups/<process group uuid>) that contains the processor component. For a user to be able to empty a queue (empty queue), they must be granted "write" in the same manor as above for "read". If you user was added to the default "all-nifi-resources" in Ranger, then they already have read and write to all NiFi policies. Effectively they are a NiFi admin user. In addition to to user being granted the ability to "read" (list queue) and "write" (empty queue), the same must be granted for all node in your NiFi cluster. This is commonly done by adding a new policy in Ranger that uses the following NiFi resource Identifier: This policy would be assigned to all nodes and and include both "read" and "write" permissions. Why is this needed? When you login in to a NIFi cluster, you are logging in to only one node. When you make a request to list a queue, you expect to see results from all nodes in your cluster. So the node you are logged in to makes a request to all nodes to return there queue list. So the originating node must be granted the ability to view the other nodes data. The same holds true when you make a request to empty a queue while logged in to one node of a cluster. That node must be able to request that the other nodes empty their queue as well. Thank you, Matt ... View more

@Hadoop User The "it'll be helpful if what processor to be used in between listenSyslog and putHDFS is suggested" question is a hard one for anyone to answer without understanding the end result you are looking for. There are The following processors: - parseSyslog (extract bits from syslog content in to FlowFile attributes) You can then use those attributes if you like to make routing decisions (routeOnAttribute), define unique target HDFS directories based on attribute value in PutHDFS - SplitText or SlitContent (Can be used to FlowFiles that contain more then one syslog message each). You get improved performance if listenSyslog ingests in batches. - UpdateAttribute (Used to add you own custom attributes or manipulate existing attributes on FlowFiles) Thanks, Matt ... View more

@Hadoop User The processor components all have tags associated to them and the associated documentation for each processor component is also embedded in the application under "help" (found in upper right corner menu). If you drag the add "Processor" icon to your canvas you will be presented with aadd processor UI. In the upper right corner is a filter box. Typing "syslog" or "hdfs" will reduce the list to those processors that share those tags. Clicking on a processor will display a brief description of the processor in the same UI near the bottom. Details documentation can be found in help or by right clicking on a processor already added to the canvas and selecting"usage" form the context menu that appears. As far as what processor you want to use depends on your complete use case. First you need to determine ho you are going to ingest this syslog data (ListenSyslog) processor is an option. As far as writing to HDFS, the PutHDFS processor is the likely choice. There are many processors available for manipulating NiFi FlowFile content between ingestion and writing out the data to a destination. Thanks, Matt ... View more

@siva karna I am not following the statement "so there is an abstraction for the first process group flow file it will stop so we will loss the data". Why would stopping a dataflow cause data loss? NiFi will only read in new nars/jars added to a NiFi lib directory on startup. There is no option to dynamically add classes during runtime. Thanks, Matt ... View more

@Akash S The ListHDFS processor records state so that only new files are listed. The processor also has a configuration option for recursing subdirectories. You could set the directory to only /MajorData/Location/ and let it list all files from the subdirectories. As new subdirectories are created, the files within those new directories will get listed. If that does not work for you, the NiFi expression language (EL) statement that you are looking for would look something like this for the directory: /MajorData/Location/${now():format('yyyy/MM/dd')} The above would cause Nifi to only look in the target directory fro Files until the day changed. I am not sure the rate at which files are written in to these target directories, but be mindful that if a file is add between runs of the listHDFS processor and the day changes between those runs, that file will not get listed using the above EL statement. Thanks, Matt ... View more

@M R I find the following very useful when trying to build Java regular expressions: http://myregexp.com The Java regular expression: ^(.*?)%%(.*?)%%(.*?)%%(.*?)%%(.*?),(.*?)%%(.*?)$ It has 7 capture groups that will result in: When you add a ew property to the extractText processor with a property name of "string" and use the above java regex. Of course if you are only looking for two capture groups, you could use the following regex instead: ^(.*?)%%.*?%%(.*?)%%.*?%%.*?,.*?%%.*?$ Thanks, Matt ... View more