@Oliver Fletcher Authentication and authorization are two separate processes within NiFi. There is no way currently for NiFi to pull LDAP groups in to its authorizer. While NiFi's file based local authorizer does support groups, those groups are not mapped to any LDAP groups. With NiFi's latest release authentication via LDAP supports only two "Identity Strategies": Identity Strategy Strategy to identify users. Possible values are USE_DN and USE_USERNAME. The default functionality if this property is missing is USE_DN in order to retain backward compatibility. USE_DN will use the full DN of the user entry if possible. USE_USERNAME will use the username the user logged in with. So either the DN returned by LDAP (USE_DN) or the username enter on the login screen (USE_USERNAME) is passed to the authorizer post any configured pattern mapping. There are currently is no Strategy for passing the user's LDAP group to the authorizer. NiFi has no support for Ranger groups as you are already aware. However, you could create a set of groups in NiFi's local file based authorizers that each provide a distinct set of access policies. You could then use your script idea to conduct ldap searches and map users DNs or usernames to those specific NiFi groups. You scripts could make calls to the nifi-api to automate adding these users to the those groups. Thanks, Matt ... View more

@bhumi limbu NiFi FlowFile attributes/metadata lives in heap. The List based processors return a complete listing from the target and then creates a FlowFile for each File in that returned listing. The FlowFiles being created are not committed to the list processor's success relationship until all have been created. So you end up running out of NiFi JVM heap memory before that can happen because of the size of your listing. As NiFi stands now, the only option is to use multiple list processors with each producing a listing of on a subset of the total files from your source system. You could use the "Remote Path", "Path Filter Regex" and/or "File Filter Regex" properties in the listSFTP to filter what data is selected to help reduce the heap usage. You could also increase the available heap to your NiFi's JVM in the bootstrap.conf file; however, I find it unlikely considering the number of FlowFiles you are listing that you will not still run out of heap memory. I logged a Jira in Apache NiFi with a suggested change to how these processors produce FlowFiles from the listing returned by these types of processors: https://issues.apache.org/jira/browse/NIFI-3423 Thanks, Matt ... View more

@Raj B Not all NiFi Processors will write attributes to FlowFiles about failures or errors. The documentation for each processor should include what attributes are written by that processor and what information those attributes will contain. There is no global enforcement by the NiFi controller on what attributes a processor must create. This is completely in the control of the developer who wrote each processor. That being said, it is good practice that any processor that has a "Failure" relationship should output an "Error" level log message that dictates the nature of the failure. This Error log message would contain the specific processor that produced the ERROR as well as information on the specific FlowFile that was routed to failure and the nature of the failure. It is possible to build a dataflow that monitors NiFi's nifi-app.log (TailFile processor) for ERROR log messages, parses out the relevant information and pass that along to some monitoring system. Thanks, Matt ... View more

@Anishkumar Valsalam Hello, During initial setup of a secured NiFi installation NiFi allows you to specify a single "initial Admin Identity". Upon first startup, NiFi will use that "Initial Admin Identity" to setup that user and grant them the "Access Policies" needed to administer that NiFi instance/cluster. That identity will be able to log in and add new users and grant "Access Policies" to those users. The default "Access Policies" that are given to that "Initial Admin Identity" include: NiFi File Based Policies: Ranger based Policies: view the UI view the user interface /flow view the controller access the controller (view) /controller (read) modify the controller access the controller (modify) /controller (write) view the users/groups access users/user groups (view) /tenants (read) modify the users/groups access users/user groups (modify) /tenants (write) view policies access all policies (view) /policies (read) modify policies access all policies (modify) /policies (write) Granting these same "Access Policies" to other users you have added will affectively make them an Admin as well. Thanks, Matt ... View more

@Saminathan A The PutSQL processor expects that each FlowFile contains a single SQL statement and does not support multiple insert statements as you have tried above. You can have the GetFile Processor route its success relationship twice with each success going to its own ReplaceText processor. Each ReplaceText processor is then configured to create either the "table_a" or "table_b" insert statement. The success from both ReplaceText processors could then be routed to the same PutSQL processor. Thanks, Matt ... View more