@Ankit13  How do you know no more files we will be put after the NiFi flow processing starts? To me in sound like the PutFile should execute at default 0 secs (as fast at it can run) and you should instead control this dataflow at the beginning were you consume the data. For example: In a 24 hour window data is being written to source directory to be consumed from between 00:00:00 and 16:00:00.  Then you want to write that data to target directory starting at 17:00.  So you instead setup a cron on a listFile processor to consume list the files at 17:00 and 17:01 and then have a FetchFile and PutFile running all the time so these immediately consume all the content for the listed files and write them to target directory.  Then your listFile does not execute again until same time next day or whatever you cron is.   This way the files are all listed at same time and the putFile can execute for as long as needed to write all those files to the target directory. Hope this helps, Matt ... View more

@Ronman  Please share the version of Apache NiFi you have installed.   Thanks, Matt ... View more

@Ankit13    Perhaps I don't understand your use case. Are you saying you have a NiFi dataflow that slowly ingested Data producing FlowFiles that work their way through your dataflow to this putFile processor? Then you want these 1000s of FlowFiles to queue up so that they can all be put to the local file system directory at the same time? So what is being suggested by @m_adeel is to use the NIPYAPI to automated the starting and stopping of the putFile processor at a given time.  You could also do the same through NiFi REST_API  calls. You would still have the challenge of when to stop it.  Does the source of data ever stop coming in?    Would you be able to put all the FlowFiles from the inbound connection queue to disk before more source FlowFiles started flowing in to the queue? Why the need to do this at a specific data and time? Thanks, Matt ... View more

@Ronman    It is interesting that you are seeing a 403 in response to trying to add a new policy on the root Process Group (PG). By granting your user string (CN=admin, OU=NIFI) to the following global policies: you should have ability to set any additional policies anywhere. From the root canvas you can either right click and select "manage access policies" or click on the small key icon in the "Operate" panel that shows the PG name and PG assigned uuid to access the PG level access policies. First you would select the policy you want to add user or group strings to from the pull-down menu. If the policy has not yet had any users assigned to it, you'll need to click on "add policy" before being able to click on the Then click on the small "add users/groups to this policy" icon to the far right. Note: Child PGs added to the canvas will by default inherit all their policies from the parent PG. You will be given option to "override" this and set child specific users/groups on that child PG.   You are stating that you encounter a 403 when you click on the "add" button? What do you see in the nifi-user.log when you perform that action? If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Ankit13    NiFi processors support "Timer Driven" and "Cron Driven" Scheduling Strategies.   Timer driven strategy allows you to specify a time interval for scheduling (for example: "30 secs" which means processor will execute every 30 seconds.). Cron Driven strategy supports a Quartz Cron [1] being used to specify when the processor should execute. There is a third option on some processors which is Event Driven that should not be used. It was created long ago and considered experimental.  It is has since been deprecated due to improvement made in the Timer Driven strategy.  It only remains in NiFi to avoid breaking flows of those who use it when they upgrade.   Important things to understand about your ask: Let's assume you configure your PutFile to execute using the Cron Driven scheduling strategy and the inbound connection to the putFile processor has multiple FlowFiles queued.  When the processor executes it will process only 1 of those FlowFiles from that inbound connection queue with default settings.  The next queued FlowFile would not get processed until the next scheduled cron execution.  While there is no way to make sure that every queued FlowFile is processed in in a single cron execution you can change the configured Run Duration: The Run Duration tells the processor to continue to use the same execution thread to execute against as many queued FlowFile as possible within the configured run duration time. Let say it takes more than 2 secs to write the very first FlowFile to the target directory.  In that case, only one FlowFile would be processed.  So there would be no perceived difference between a run duration of 0ms and 2s. In a NiFi cluster, each node in the cluster executes the dataflow against the FlowFiles queued on that same node.  So if FlowFiles were queued on the inbound connection to PutFile on all nodes, each node would execute 1 at each cron interval processing through FlowFile(s) per node as described above.   [1] https://community.cloudera.com/t5/forums/replypage/board-id/Questions/message-id/229905   If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

Hello @edoS  Welcome to the community! NiFi's provides so many option for user authentication and authorization, setting up exactly what you need can be overwhelming at times. This is certainly something the Cloudera support could walk you through if you have a support contract with us that covers the NiFi service. At a high level, here is what you need to understand about the authentication and authorization process in NiFi. Authentication happens first and must be successful before any authorization is verified.  NiFi supports numerous ways to authenticate users/clients (TLS, Kerberos, LDAP, openID, etc...).  No matter which method is used, the end result of any authentication is a user string that identifies the successfully authenticated user/client. That user string is then evaluated against the identity mappings [1] you may have configured in the nifi.properties file.  These identity mappings are used to normalize the user strings. for example: Trim the CN from the full DN in a user/client certificate Trim the user name from a kerberos principal convert the user string to all uppercase or lowercase The resulting user/client string is then passed to the authorizer to verify that user/client is authorized for the NiFi Resource Identifier being requested. NiFi authorizers.xml is where this configuration is setup.  This file is easiest to read from the bottom up.   At the bottom of the authorizers.xml you will find your authorizer which you have setup as the "Ranger-Provider".  It is important to understand how this authorizer works.  NiFi runs a background thread that checks in with Ranger to see if there is a new policy definition for the NiFi service.  If so, the new definition is downloaded by NiFi. What Ranger provides to NiFi in this downloaded policy definition are all the polices setup in Ranger.  For each there will be the "NiFi Resource Identifier(s)" along with the user strings and group strings that have been assigned "Read" and/or "Write" permissions.  Now remember up to this point all NiFi knows about the authenticated user is the user string.  NiFi has no idea yet what groups that user string may belong to.  Within the Ranger-Provider, you will find a property name with "User Group Provider". The value set here tell the authorizer where to check to see if the user string passed from authentication has any known user to group associations. Search your authorizers.xml for configured User Group Provider [2].  There are numerous options that can be configured for determining user to group associations.  Some of the available providers allow you to configured multiple providers.  While the authorizer "ranger-provider" can only point at 1, it may point at a "composite-configurable-user-group-provider" [3] for example that can be setup to reference multiple user-group-providers.  The key here is making sure you have added 1 or more user group providers that will return all the user to group associations you need.  Based on the log output you shared from the nifi-user.log. We know that none of the user group providers you may have setup returned any group strings associated to your user string (identity[18330301],groups[] ).  This is why "groups [ ]" is empty. The "file-user-group-provider" [4] allows you to create user string to group string associations manually via the NiFi UI directly.  The commonly used "ldap-user-group-provider" [5] determines user and group associations via user and/or group syncs with ldap/AD. Now that NiFi knows what groups the authenticated user string is associated with, the user and the groups can be checked against the downloaded policies to see if the user is authorized for the action being performed or the end-point trying to be accessed.   [1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties [2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizers-setup [3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#composite-implementations [4] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#fileusergroupprovider [5] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldapusergroupprovider   If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more