@jirungaray  Cloudera Flow Management (Based on Apache NiFi) provides multiple methods for managing user authorization.  This includes NiFi internally via the File-Access-Policy-Provider and externally via Apache Ranger.  There is no built in mechanism for auto setting up authorization policies for users or groups with the exception of the Initial Admin and Initial NiFi Node authorizations. Many of the Authorization policies are directly related to the components added to the canvas.  Those components are assigned unique IDs making it impossible to  create policies before the components exist.  File-Access-Policy-Porvider:  This provider utilizes a file on disk (authorizations.xml) to persists authorization policies.  This file is loaded when NiFi starts.  This means it is possible to manually generate this file and have NiFi load it on startup.   Also as you mentioned, you could script out the authorization creating through NiFi Rest-API calls.  Ranger provider: This moves authorization responsibility over to Apache Ranger.  Policies setup within Ranger are download by the NiFi nodes where they are locally enforced.  No matter which authorizer you choose to use, authorizations are easiest to manage via groups.  Typical users setup ldap groups for various NiFi roles (admins, team 1, team2, etc..) and makes specific users members of these groups.  This simplifies authorization since you can authorizer these groups instead of the individual users. Simply adding or removing a user as member of one of these authorized groups gives or removes authorized access to the NiFi resource identifier (NiFi policy).  The ldap-user-group-provider can be added to the NiFi authorizers.xml to auto manage syncing of user and group identities from your AD/LDAP further simplifying management over the file-user-group-provider method which requires the manual adding of user and group identifiers to the NiFi. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@tono425  I assumed all your records were of the same schema.  With Merge Record, a bin will consist of potentially many 'like FlowFiles'. In order for two FlowFiles to be considered 'like FlowFiles', they must have the same Schema (as identified by the Record Reader).  If a FlowFile is not like other FlowFiles already allocated to a bin, it will be allocated to a different bin.  I would still recommend against using min record to 1 since a typical dataflow will be a constant stream of new FlowFiles and the MergeRecord processor will only see those FlowFiles queued at the exact moment of execution. So you can result in smaller then expected number of records in a merge Record with a constantly running dataflow.   Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Chram  Lets backup and first understand the use case you are trying to solve here. From the look of your dataflows it appears you have two FlowFiles that you want to merge via the MergeContent processor.  The order in which they are merged is important to you.  This sounds like a use case better solved by using the "Defragment" merge strategy in the MergeContent processor rather then trying to use EnforceOrder processor. Is there more to your use case which necessitates the use of enforceOrder? As far as your dataflow goes,  I am having trouble reproducing the issue you described. I see my priority 2 file sitting in wait only until my matching priority 1 file arrives.  The dataflow screenshot you shared also does not show a wait relationship being routed anywhere.  Are you using the "retry" + "terminate" checkboxes on the wait relationship instead?  If so, what are the retry settings configured? Rather then use "retry" on the wait relationship, try dragging a new connection away from and back to the EnforceOrder processor to create a loop and assign the "Wait" relationship to that looped connection (make sure to also uncheck "retry" on the wait relationship).   Does same issue still persist? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more