@JorgeO    Looking at your nifi-user.log output we see the caused by line as: Caused by: org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result size: expected 1, actual 2   This means that your LDAP/AD returned two results instead of only one.  NiFi then has not idea which of those returns is the correct one it should be using. So this either an issue within your LDAP/AD or an issue within your current login-identity-providers.xml filters.  But your last shared file looks fine to me. I would suggest using the ldapsearch command to run a ldap query outside of NiFi to see what returns you get for your admin user (cn=<admin user>) Hope this helps, Matt ... View more

@JorgeO    I am not clear on what "nifi screen does not show any change" means. Some things to keep in mind: 1. NiFi must first be secured before NiFi will even support any form of user authentication or authorization. http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_properties 2. Authentication and authorization processes are handled independently. A user must first successfully authenticate and only then will the user identity be passed to the authorization process to verify authorized policies granted for that user.  3. The ldap-provider does not sync users from ldap, it is only used to validate user credentials supplied at a login prompt with ldap to authenticate that user.  Are you seeing a login window with your NiFi?  (will not see this if NiFi is not yet secured). 4. Once your user successfully authenticates then authorization comes in to play.  Looking at your authorizers.xml, one issue stands out to me: - You configured:   <property name="Initial Admin Identity">cn=admin,dc=example,dc=local</property>   This is set in the "file-access-policy-provider".  This provider is used to seed the initial minimum required policy that an admin user would need in order to access the UI, add additional users to the authorizer through UI and associate those additional users to various NiFi authorization policies. - If you are a multi-node NiFi cluster, the node certificate DNs for each of yoru secured nodes also need to be configured in the "file-access-policy-provider":   <property name="Node Identity 1">CN=node1, OU=nifi</property> <property name="Node Identity 2">CN=node2, OU=nifi</property> etc...   - Problem here is that I don't see where you create that initial admin user.  NiFi can not seed policies for a user that does not exist yet as a known identity to NiFi.  Multiple methods are available for adding users and groups to NiFi for policy assignment. Your setup shared is using the "file-user-group-provider" which uses locally defined user/client identities.  However, you did not add your admin user DN to the file-user-group-provider so that NiFi adds it to the users.xml.  It needs to exist before the "file-access-policy-provider" can associate policies to that user identity.   <property name="Initial User Identity 1">cn=admin,dc=example,dc=local</property>   - Additionally, if you have setup a NiFi multi-node cluster, the DNs from the certificate used to secure each of the nodes must also be added as "Initial User Identity 2,3,4..." properties in the file-user-group-provider also. All authentication and authorization actions would be logged in the nifi-user.log.  So check that to see what is happening if you are having access or authorization exceptions.  If this log does not exist or is empty, odds are that your NiFi was not secured.   In addition to the security properties I provided the link to above, you also need to make sure you set the following additional properties in the nifi.properties: nifi.web.https.host The HTTPS host. It is blank by default. nifi.web.https.port The HTTPS port. It is blank by default. When configuring NiFi to run securely, this port should be configured. The "nifi.web.https.port" is essentially the switch that enables secured NiFi when set. The default is "nifi.web.http.port" which is a non-secured port. Hope this helps, Matt ... View more

@adhishankarit  The issues is being caused by the line returns used in the middle of the NiFi NiFiExpression Language (EL) ifElse() function you are using. The text box where you enter your NiFi EL uses a NiFi editor that highlights to show proper EL format.  You'll notice your EL stops highlighting once you reach first line return. So you'll notice character 32 is the first single quote character.  Since EL breaks at this point it fails to find the matching second expected single quote. This leaves you with two options: 1. Create flat json without the line returns. 2. Looking at result you are trying to achieve, design your NiFi EL differently: Note proper NiFi EL highlighting above. Hope this helps, Matt ... View more

@Lallagreta  Make sure you do not have any line returns in the values for your dynamic properties added in the UpdateAttribute processor.  When you click on the value field for each property you should not see a line "2". For example: Above would result in the value assigned to the FlowFile Attribute having a line return.  If this is the case, edit the properties value(s) to remove the line returns so you only see one line (1). Hope this helps, Matt ... View more

@Siddo  The current strategy you are using is the best option with a use a case where the client is sending/pushing data to listeners across your NiFi cluster nodes.  Whenever you have a client that is pushing data to NiFi, this setup avoids as your mentioned having a single point of failure.  If a load balancer can't be used, It becomes the responsibility of the client to detect delivery problems and switch to delivering to a different node. Load balancing within NiFi's dataflows is the best option when the dataflow is consuming from a source system.  Some data consumption methods are not cluster friendly (for example FTP).   This is because every node in a NiFi cluster executes the same flow.xml.gz.  If you had for example the listSFTP/GetSFTP processors running on every node, you would have data duplication and potentially issues as every node tried to consume the same data.  So in this scenario you would configure the processor to execute on the primary node only and then use LB connections to immediately redistribute those FlowFiles across your cluster before doing further processing.  This is why we created the List and Fetch processor pairs.  These are typically non cluster friendly type processors.  So a ListSFTP produces FlowFiles with zero content and only attributes with details on where to fetch a specific FlowFiles content.  Those 0 bytes FlowFiles quickly Load balance across the cluster where the FetchSFTP processor would fetch the actual content for the FlowFile specific data file and insert it into the FlowFile.   This type of setup also avoid single point of failure since loss of the currently elected primary node (where the data lister/consumer is running) would result in a new node being elected as the new Primary node.  That new primary node reads state from the cluster state provider and begins listing where the previous elected node's list processor stopped. So you can see that each use case has very specific benefits/use cases. Another scenario may be that even with an external F5 LB, you may find one node in your cluster ends up with a larger burden of work load (maybe one node ends up with bulk of larger data files.  That data can be redistribute on connections were such single node bottle necks occur to re-balance the load at that point in a dataflow.  So at times a combination may make sense as well, but I would not just apply this strategy unless needed since it adds to network usage. NiFi's internal LB connections can also be used to move all data to a single node for some use case.  Let's say there is a batch of data spread out across multiple NiFi nodes that you want to merge in to a single FlowFile.  NiFi nodes each work on only the FlowFiles on their own node.  But using LB connection in specific spots on your flow would allow you to move all like data to the sam node before a merge type processor. Hope this helps, Matt ... View more

@Lyoung    The NiFi client (NiFi or MiNiFi instance running the Remote Process Group (RPG)) has not control over the connection with the server (NiFi configured with Remote input or Output ports). The RPG is provided with a http or https address of one or more target NiFi nodes in a NiFi cluster). A background thread connects to that target NiFi to fetch Site-To-Site (S2S) details. If the target is https enabled, a mutual TLS handshake will happen. This means the client must have a keystore and truststore configured in the nifi.properties (NiFi) or config.yaml (MiNiFi) that can successfully be used to mutually authenticate with the target NiFi server. The server side NiFi must have the properties you listed configured: nifi.remote.input.host=<must be set to hostname of NiFi on which you are configuring this property. This is the hostname returned to client in the S2S details.  Be careful that what ever you set heer does not resolve to localhost.> nifi.remote.input.secure=false (this tells client if connection is secure or unsecure. If false, the "nifi.web.http.port" property must be set and the URL used in the RPG must be  "http://<target nifi>:<http port>/nifii".  If set to true, the "nifi.web.https.port" property must be set and the URL used in the RPG must be  "https://<target nifi>:<https port>/nifii" nifi.remote.input.socket.port=<This is the RAW port that will be used to actually send or receive the FlowFiles from remote Input or Output ports on target NiFi node(s).  If this property is not set on the target NiFi node(s), RAW transport protocol will not be supported. (S2S details are always fetched over HTTP)> nifi.remote.input.http.enabled=true. <This properties states whether the "http" transport protocol can be used for sending the FlowFiles.> nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs   Based on the log output shared it sounds like above properties were not set on the Target NiFi node(s). Did you set them on client NiFi (NiFi actually running the RPG)? In addition to the Target NiFi S2S details above for each target NiFi node being returned to client, the details will also include the FlowFile load on each node, Remote input ports that client has been authorized to use, and Remote Output ports that the client has been authorized to use.  If the target server side NiFi node(s) are unsecured then there will be no authorization set for ports, all clients would have access to all remote input/output ports. Also keep in mind that any changes to NiFi's/MiNiFi's configuration files would require a restart of the service before they would be applied. Aside from above, I would need to see screenshots and nifi.properties/config.yaml configs of both your  client and server side of this S2S connection to help further. Hope this helps, Matt ... View more

@garoosy  You should look in to using the "ExecuteSQLRecord" instead of "ExecuteSQL" for large volume data.  To be efficient here you  would have many records in a single FlowFile. Right now you have a single record per each FlowFile which is not going to be very efficient.  The only way for "ExecuteSQL" to handle multiple FlowFile executions in a single connection is if the SQL statement used in every FlowFile is identical.  In order to do that the unique values would need to come from FlowFile attributes. You may find these post helpful: https://community.cloudera.com/t5/Support-Questions/Nifi-ExectueSQL-how-to-force-a-parameter-to-be-a-string/td-p/240117 https://stackoverflow.com/questions/63330790/using-nifi-executesqlrecord-with-parameterized-sql-statements   If you have threads that never seem to complete (will see small number in upper right corner of processor (2)), it is best to get a series of thread dumps (4 - 6) to verify thread is not progressing.  Then you have to determine if what the thread is waiting on.  Did you try setting a "Max Wait Time" on the processor?  It defaults to 0 which means it would wait forever.   Hope this helps, Matt ... View more

@adhishankarit    As I mentioned you need an additional unique attribute that you only add on the failure path (ConstructHDFSError UpdateAttribute) to MergeContent. overall-status = ERROR Since this attribute (overall-status) is not being set on the success path, the mergeContent "Attribute strategy" set to "Keep All Unique Attributes" will then set this overall-status attribute on the merged FlowFile produced. Keep All Unique Attribute --> any attribute on any FlowFile that gets bundled will be kept unless its value conflicts with the value from another FlowFile. Since you are not setting this attribute on your success path FlowFiles, it would only be set on mergeFlowFiles where one of more FlowFiles traversed the failure flow path.  This allows you to capture overall-status of the zip bundle. Then in your ReplaceText processor you would use a more complex NiFi Expression Language (EL) in your replacement value. Something like: ${uniquefile}:${overall-status:isNull():ifElse('success','${overall-status}')}:${message}   This will set "success" if the "overall-status" attribute does not exist on any FlowFiles that were part of the merged FlowFiles; otherwise it will set the it to the value set in the "overall-status"  attribute. If you found this help, please take a moment to click "accept solution" on all responses that helped. Matt   ... View more