@bsivalingam83    The ability to "ignore" properties in various NiFi config files was added with the CFM 1.0.1 release.   With older CFM versions (1.0.0) you can set a safety valve to overwrite the current set java.arg.13 value with something else: Above simply defines a key=value pair which would simply not be used by NiFi bootstrap.  End result, NiFi no longer using G1GC and instead using the default Garbage Collector for your version of Java being used. Hope this helps, Matt   ... View more

@Jarinek  The process really depends on what update you are trying to make. 1. You can not remove a connection that has queued FlowFiles in it, but you can redirect it to a different target processor with queued data. 2. You can not redirect a connection if the processor it is currently attached to still has a running thread.  Stopping a processor does not kill threads, it simply tells the processor to not execute again at the configured run schedule.  Existing threads will continue to run until they complete.  Until all threads exit, the processor is still in a state of "stopping" even though UI reflect red square for "stopped". 3. You cannot modify a processor if is still has running threads (see note about "stopping" processors above) 4. If you stop the component that is on the receiving side of a connection, any FlowFiles queued on that connection, not tied to any active thread still running on target processor component, will not be processed and remain queued on the connection.  You can manual empty a queue through a rest-api call (means data loss), but that is not necessary if you are not deleting the connection. Attempts to perform configuration changes when components still have active threads or are in a running state will result in an exception being thrown and the change not happening. Attempts remove connections that have queued FlowFiles will throw an exception and block removal. Now if all you are trying to do is modify some configuration on a processor, all you need to do is stop the processor, check that it has no active threads, make the config change, and then start the processor again. Not sure wha you are asking with " update the flow ignoring any data in failure or error connection queues".  NiFi does not ignore queued FlowFiles.  It also not wise to leave connection with queued FlowFiles just sitting around your dataflows.   Those old queued FlowFile will prevent removal or content claims that contain that FlowFiles data.  Since a content claim can contain the data from 1 to many FlowFiles, this can result in your content repository filling up.  NiFi can only remove content claims which have no FlowFiles pointing to them anymore. Here are some useful links: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nipyapi.html https://github.com/Chaffelson/nipyapi http://nifi.apache.org/docs/nifi-docs/rest-api/index.html https://community.cloudera.com/t5/Community-Articles/Update-NiFi-Connection-Destination-via-REST-API/ta-p/244211 https://community.cloudera.com/t5/Community-Articles/Change-NiFi-Flow-Using-Rest-API-Part-1/ta-p/244631 Hope this helps, Matt ... View more

@medloh  The article you are using for reference is old and bit out of date. As part of the  work that went into NIFI-3921 the schema properties within the putParquet processor were removed.  Before these changes were made at the time of that article you referenced, you had to set the schema properties and they had to match the schema properties set in the recordReader.   With the changes the processor simply gets them from the reader so they do not need to be configured a second time in the processor properties. Also at time of that article there was no parquetReader or ParquetRecordSetWriter controller services.  Now that NiFi has a ParquetReader and writer, you can use the ConvertRecord processor to read a source FlowFiles and convert it parquet within your dataflow and then have freedom to use whatever processor you want downstream in your dataflow to write out the parquet content.  You can think of the putParquet as a combination of ParquetRecordSetWriter and putHDFS with a selectable recordReader only. Hope this helps, Matt ... View more

@BhaveshP  I am in complete agreement with @tusharkathpal response. But you should be able to work around this issue through a configuration change in your nifi.properties file. nifi.web.proxy.host=dev.example.com:<port number> Property description:  A comma separated list of allowed HTTP Host header values to consider when NiFi is running securely and will be receiving requests to a different host[:port] than it is bound to. For example, when running in a Docker container or behind a proxy (e.g. localhost:18443, proxyhost:443). By default, this value is blank meaning NiFi should only allow requests sent to the host[:port] that NiFi is bound to. Since the hostname your client is using does not match any SAN in the individual nodes certificates, the above property allows NiFi to accept this additional hostname. The other option is to create new certificates for each of your NiFi nodes where "dev.example.com' is added as an additional SAN entry. Hope this helps, Matt   ... View more

@Umakanth    Any chance you are running a NiFi cluster (multiple NiFi nodes) or you have multiple systems all trying to consume the same data from this same SFTP server? It is possible that one host finished reading the file first and removed it before the other hosts could finsih reading the same file. SFTP is not a cluster friendly protocol and if using this processor in a NiFi Cluster, this processor should be configured to execute on "primary node" only.  Otherwise all nodes in your cluster will be fighting to consume the same source files of which you can expect to see exceptions. The GetSFTP processor is also a deprecated processor in favor of the newer listSFTP and FetchSFTP set of processors.  The newer processors allow you listSFTP (primary node only and produces 0 byte FlowFiles) ---> load balanced connection (balances FlowFiles across all nodes in cluster)--> FetchSFTP ( Execute on all nodes. Retrieves specific content per FlowFile). Hope this helps, Matt ... View more

@Arash    In your 4 node NiFi cluster, what value do you have set in the " nifi.remote.input.host" property in the nifi.properties file for each of the 4 nodes?  It should be the FQDN for each node and not be the same value on all 4 nodes. Form the host where MiNiFi is running, can all 4 of those FQDNs be resolved and reachable over the network?  If not, MiNiFI RPG is only going to be able to send successfully to one FQDN it can reach. When the RPG is started it reaches out to the URL configured in the RPG to obtain S2S details from the target host.  That target host collects the host details for all currently connected nodes in the cluster and communicates that back to the client (MiNiFi).   If all 4 nodes report the same configured FQDN in the "nifi.remote.input.host" property, then client only knows of one FQDN to which it can send FlowFiles over Site-To-Site (S2S). To improve redundancy in the RPG, you can provide a comma separated list of URLS in the RPG configuration so if any one node is down, the RPG can try fetch S2S details from the next host in the comma separated list. Hope this helps, Matt ... View more

@JorgeO    Looking at your nifi-user.log output we see the caused by line as: Caused by: org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result size: expected 1, actual 2   This means that your LDAP/AD returned two results instead of only one.  NiFi then has not idea which of those returns is the correct one it should be using. So this either an issue within your LDAP/AD or an issue within your current login-identity-providers.xml filters.  But your last shared file looks fine to me. I would suggest using the ldapsearch command to run a ldap query outside of NiFi to see what returns you get for your admin user (cn=<admin user>) Hope this helps, Matt ... View more

@JorgeO    I am not clear on what " nifi screen does not show any change" means. Some things to keep in mind: 1. NiFi must first be secured before NiFi will even support any form of user authentication or authorization. http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_properties 2. Authentication and authorization processes are handled independently. A user must first successfully authenticate and only then will the user identity be passed to the authorization process to verify authorized policies granted for that user.  3. The ldap-provider does not sync users from ldap, it is only used to validate user credentials supplied at a login prompt with ldap to authenticate that user.  Are you seeing a login window with your NiFi?  (will not see this if NiFi is not yet secured). 4. Once your user successfully authenticates then authorization comes in to play.  Looking at your authorizers.xml, one issue stands out to me: - You configured:   <property name="Initial Admin Identity">cn=admin,dc=example,dc=local</property>   This is set in the "file-access-policy-provider".  This provider is used to seed the initial minimum required policy that an admin user would need in order to access the UI, add additional users to the authorizer through UI and associate those additional users to various NiFi authorization policies. - If you are a multi-node NiFi cluster, the node certificate DNs for each of yoru secured nodes also need to be configured in the "file-access-policy-provider":   <property name="Node Identity 1">CN=node1, OU=nifi</property> <property name="Node Identity 2">CN=node2, OU=nifi</property> etc...   - Problem here is that I don't see where you create that initial admin user.  NiFi can not seed policies for a user that does not exist yet as a known identity to NiFi.  Multiple methods are available for adding users and groups to NiFi for policy assignment. Your setup shared is using the "file-user-group-provider" which uses locally defined user/client identities.  However, you did not add your admin user DN to the file-user-group-provider so that NiFi adds it to the users.xml.  It needs to exist before the "file-access-policy-provider" can associate policies to that user identity.   <property name="Initial User Identity 1">cn=admin,dc=example,dc=local</property>   - Additionally, if you have setup a NiFi multi-node cluster, the DNs from the certificate used to secure each of the nodes must also be added as "Initial User Identity 2,3,4..." properties in the file-user-group-provider also. All authentication and authorization actions would be logged in the nifi-user.log.  So check that to see what is happening if you are having access or authorization exceptions.  If this log does not exist or is empty, odds are that your NiFi was not secured.   In addition to the security properties I provided the link to above, you also need to make sure you set the following additional properties in the nifi.properties: nifi.web.https.host The HTTPS host. It is blank by default. nifi.web.https.port The HTTPS port. It is blank by default. When configuring NiFi to run securely, this port should be configured. The "nifi.web.https.port" is essentially the switch that enables secured NiFi when set. The default is "nifi.web.http.port" which is a non-secured port. Hope this helps, Matt ... View more

@adhishankarit  The issues is being caused by the line returns used in the middle of the NiFi NiFiExpression Language (EL) ifElse() function you are using. The text box where you enter your NiFi EL uses a NiFi editor that highlights to show proper EL format.  You'll notice your EL stops highlighting once you reach first line return. So you'll notice character 32 is the first single quote character.  Since EL breaks at this point it fails to find the matching second expected single quote. This leaves you with two options: 1. Create flat json without the line returns. 2. Looking at result you are trying to achieve, design your NiFi EL differently: Note proper NiFi EL highlighting above. Hope this helps, Matt ... View more

@Siddo  The current strategy you are using is the best option with a use a case where the client is sending/pushing data to listeners across your NiFi cluster nodes.  Whenever you have a client that is pushing data to NiFi, this setup avoids as your mentioned having a single point of failure.  If a load balancer can't be used, It becomes the responsibility of the client to detect delivery problems and switch to delivering to a different node. Load balancing within NiFi's dataflows is the best option when the dataflow is consuming from a source system.  Some data consumption methods are not cluster friendly (for example FTP).   This is because every node in a NiFi cluster executes the same flow.xml.gz.  If you had for example the listSFTP/GetSFTP processors running on every node, you would have data duplication and potentially issues as every node tried to consume the same data.  So in this scenario you would configure the processor to execute on the primary node only and then use LB connections to immediately redistribute those FlowFiles across your cluster before doing further processing.  This is why we created the List and Fetch processor pairs.  These are typically non cluster friendly type processors.  So a ListSFTP produces FlowFiles with zero content and only attributes with details on where to fetch a specific FlowFiles content.  Those 0 bytes FlowFiles quickly Load balance across the cluster where the FetchSFTP processor would fetch the actual content for the FlowFile specific data file and insert it into the FlowFile.   This type of setup also avoid single point of failure since loss of the currently elected primary node (where the data lister/consumer is running) would result in a new node being elected as the new Primary node.  That new primary node reads state from the cluster state provider and begins listing where the previous elected node's list processor stopped. So you can see that each use case has very specific benefits/use cases. Another scenario may be that even with an external F5 LB, you may find one node in your cluster ends up with a larger burden of work load (maybe one node ends up with bulk of larger data files.  That data can be redistribute on connections were such single node bottle necks occur to re-balance the load at that point in a dataflow.  So at times a combination may make sense as well, but I would not just apply this strategy unless needed since it adds to network usage. NiFi's internal LB connections can also be used to move all data to a single node for some use case.  Let's say there is a batch of data spread out across multiple NiFi nodes that you want to merge in to a single FlowFile.  NiFi nodes each work on only the FlowFiles on their own node.  But using LB connection in specific spots on your flow would allow you to move all like data to the sam node before a merge type processor. Hope this helps, Matt ... View more

@Lyoung    The NiFi client (NiFi or MiNiFi instance running the Remote Process Group (RPG)) has not control over the connection with the server (NiFi configured with Remote input or Output ports). The RPG is provided with a http or https address of one or more target NiFi nodes in a NiFi cluster). A background thread connects to that target NiFi to fetch Site-To-Site (S2S) details. If the target is https enabled, a mutual TLS handshake will happen. This means the client must have a keystore and truststore configured in the nifi.properties (NiFi) or config.yaml (MiNiFi) that can successfully be used to mutually authenticate with the target NiFi server. The server side NiFi must have the properties you listed configured: nifi.remote.input.host=<must be set to hostname of NiFi on which you are configuring this property. This is the hostname returned to client in the S2S details.  Be careful that what ever you set heer does not resolve to localhost.> nifi.remote.input.secure=false (this tells client if connection is secure or unsecure. If false, the "nifi.web.http.port" property must be set and the URL used in the RPG must be  "http://<target nifi>:<http port>/nifii".  If set to true, the "nifi.web.https.port" property must be set and the URL used in the RPG must be  "https://<target nifi>:<https port>/nifii" nifi.remote.input.socket.port=<This is the RAW port that will be used to actually send or receive the FlowFiles from remote Input or Output ports on target NiFi node(s).  If this property is not set on the target NiFi node(s), RAW transport protocol will not be supported. (S2S details are always fetched over HTTP)> nifi.remote.input.http.enabled=true. <This properties states whether the "http" transport protocol can be used for sending the FlowFiles.> nifi.remote.input.http.transaction.ttl=30 sec nifi.remote.contents.cache.expiration=30 secs   Based on the log output shared it sounds like above properties were not set on the Target NiFi node(s). Did you set them on client NiFi (NiFi actually running the RPG)? In addition to the Target NiFi S2S details above for each target NiFi node being returned to client, the details will also include the FlowFile load on each node, Remote input ports that client has been authorized to use, and Remote Output ports that the client has been authorized to use.  If the target server side NiFi node(s) are unsecured then there will be no authorization set for ports, all clients would have access to all remote input/output ports. Also keep in mind that any changes to NiFi's/MiNiFi's configuration files would require a restart of the service before they would be applied. Aside from above, I would need to see screenshots and nifi.properties/config.yaml configs of both your  client and server side of this S2S connection to help further. Hope this helps, Matt ... View more