@littlesea374    The WAIT processor requires a release signal that is typically created using the NOTIFY processor. So that really will not help here. Perhaps you could try setting a penalty on each FlowFile.  Penalized FlowFiles are not processed by the follow on processor until the penalty duration has ended.  This can be done using the ExecuteScript processor after listSFTP:   You then set length of "Penalty Duration" via the settings tab.  Set penalty high enough to ensure the file writes have completed.  Of course this does introduce some latency. What this will not help with is listSFTP still listing the same files multiple times.  As data is written, the timestamp written on that source FlowFile updates, which means it will get listed again as if it is a new file. but the delay here allows full data to be written and then perhaps you can use a detectDuplicate processor to remove duplicates based on filename before you actually fetch the content. just some thoughts here, but that JIra is probably best path....   Matt ... View more

@littlesea374    The ListSFTP processor's only mechanism for ignoring files in the listing directory are those marked as hidden (start with . on linux based systems).   The dot rename of file transfer is pretty common with SFTP. Now if the files are being streamed into the SFTP server by another processes that does not use some form of dot rename or filename change, you would need the new feature added to listSFTP as part of https://issues.apache.org/jira/browse/NIFI-5977   This new feature is part of Apache NiFi 1.10 which adds a couple new configuration properties to the listSFTP processor.  Minimum File Age is what you would need to use.  Only files where the last update time is older than this configured value would be listed. Hope this helps, Matt ... View more

@stevenmatison    NiFi that comes with HDF is never exactly the same as the Apache NiFi version.  It is simply based off an Apache NiFi release with some modifications and in many cases additional fixes not part of the Apache release. There is currently no plan for an HDF release based off Apache NiFi 1.10.0.  That may change. The current to plan is for Apache NiFi 1.10 to show up in the Cloudera Data Platform - Data Cloud (CDP-DC) release.   Thank you, Matt     ... View more

@Cl0ck    The basics: 1. NiFi must be secured before any form of Authentication and authorization can be utilized 2. You can not have a mix of secured and unsecured nodes in a single NiFi cluster.  Unsecured NiFi nodes would not be able to communicate with the secured NiFi nodes. 3. Securing NiFi requires that you provide each NiFi node with a keystore file that contains a single PrivateKeyEntry that includes at a minimum the following: --- ExtendedKeyUsage supporting both clientAuth and serverAuth --- SubjectAlternativeName(s) that match the exact hostname for the NiFi node that keystore is being installed on. --- The keystore and key passwords need to be the same. 4. Securing NiFi also requires a truststore file which includes all the trust authorities.  These TrustedCertEntries can be the public certs of the root and intermediate Certificate Authorities (CAs) or the public cert for any self signed certs you may have created.   Note: The NiFi CA makes it easy to setup a CA and sign certificates for your nodes; however, it is not a full featured CA and is not recommended for production use. Securing NiFi requires the setting of the following properties in the nifi.properties file: nifi.security.keyPasswd= nifi.security.keystore=/<path to>/keystore.jks nifi.security.keystorePasswd= nifi.security.keystoreType=JKS nifi.security.truststore=/<path to>/truststore.jks nifi.security.truststorePasswd= nifi.security.truststoreType=JKS nifi.web.https.host=<hostname> nifi.web.https.network.interface.default= nifi.web.https.port=<secure port> nifi.cluster.protocol.is.secure=True nifi.security.user.authorizer= (optional) nifi.security.user.login.identity.provider= Once NiFi is configured to be secure, you need to consider how your users with authenticate: By default once secured, NiFi will require that all users authenticate via client/user TLS certificates. However, NiFi offers a variety of additional authentication methods that can be configured as additions to the TLS authentication. 1. TLS authentication (always attempted first) 2. Spnego (configured in nifi.properties and attempted second if configured) 3. Login provider (configured in the login-identity-providers.xml.  Option include LDAP, kerberos, knox, and OpenID connect. Can only configure one and this is attempted third if neither one or two resulted in a client authenctication) You mentioned that you want to use LDAP, so you would need to configure the ldap-provider in the login-identity-providers.xml and set the property "nifi.security.user.login.identity.provider=ldap-provider" in the nifi.properties file.   After authentication comes authorization... This is what the now authenticated user is allowed to do/access within your NiFi.  Authorization configuration is done via the authorizers.xml file. It is easiest to read this file from the bottom up. at the bottom you should have an "authorizer": <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> Above is for the managed-authorizer which you would then reference in the nifi.properties file using the property "nifi.security.user.authorizer=managed-authorizer"   You can see this authorizer calls a "file-access-policy-provider" which you must find above this entry in the authorizers.xml: <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">composite-user-group-provider</property> <property name="Node Group"></property> <property name="Initial Admin Identity"><username of ldap user who will act as the initial admin user></property> <property name="Authorizations File">/<path to>/authorizations.xml</property> <property name="Node Identity 2"><dn of node 1 in cluster></property> <property name="Node Identity 3"><dn of node 2 in cluster></property> <property name="Node Identity 4"><dn of node N in cluster></property> </accessPolicyProvider> The above is responsible for setting up some initial necessary authorizations for your initial admin user and NiFi nodes in your cluster.  You can see this provider calls another provider "composite-user-group-provider" which you will find further up in the authorizers.xml.  This provider builds the authorizations.xml file.   Note: Authorizations.xml file is only created if it does not already exist.  If it exists already, no changes will be made to it if you modify this file.  Expectation is that all new authorizations are made via the NiFi UI by your initial admin user. The next three providers control where the NiFi authorizer learns about the users to which authorizations will be granted. <userGroupProvider> <identifier>composite-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="User Group Provider 1">ldap-user-group-provider</property> <property name="Configurable User Group Provider">file-user-group-provider</property> </userGroupProvider> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">/<path to>/users.xml</property> <property name="Initial User Identity 2"><dn of NiFi node 1></property> <property name="Initial User Identity 3"><dn of NiFi node 2></property> <property name="Initial User Identity 4"><dn of NiFi node 3></property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN"><dn of ldap manger></property> <property name="Manager Password"><ldap manager password></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://<hosntame>:<port></property> <property name="Page Size">500</property> <property name="Sync Interval">30 mins</property> <property name="User Search Base"></property> <property name="User Object Class"></property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter"></property> <property name="User Identity Attribute"></property> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base"></property> <property name="Group Object Class"></property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute"></property> <property name="Group Member Attribute"></property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> Of course you will need to fill in all the required ldap properties to sync you users and groups from your LDAP.  The file-user-group-provider is used for creating any local users.  You will see above it created your NiFi nodes as local users in a users.xml file since it is very unlikely these NiFi servers will exist in ldap.   Note; The users.xml is only generated once.  If it already exists it will not be updated with any future changes made to this file.  Expectations are that additional local users are added by your initial admin manually via the NiFi UI.   Note: Always use search filters to limit number of users and groups returned to only those that will be accessing your NiFi.  NiFi holds all these returned users/groups in heap memory, so you want to avoid syncing the entire ldap.   Other things to keep in mind... 1. NiFi is case sensitive.  User "John Smith" is not the same as user "john smith". 2. The user string that results from successful authentication must match EXACTLY" with the user string returned by the ldap-user-group-provider or file-user-group-provider. NIFi identity mapping patterns can be used to trim/modify the strings returned by both the authentication provider and authorization providers so they match: examples: nifi.security.group.mapping.pattern.anygroup=(.*?) nifi.security.group.mapping.value.anygroup=$1 nifi.security.group.mapping.transform.anygroup=LOWER nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?)$ nifi.security.identity.mapping.value.dn=$1 nifi.security.identity.mapping.transform.dn=LOWER nifi.security.identity.mapping.pattern.kerb=^(.*?)@(.*?)$ nifi.security.identity.mapping.value.kerb=$1 nifi.security.identity.mapping.transform.kerb=LOWER   Hope this helps you get started, Matt   ... View more

@ChampagneM12    When you install a NiFi cluster, you start with a blank canvas.  So there is no data ingestion at first.  The  user must construct data flow(s) to meet their individual use cases as I am sure you know.  Handling data ingestion through an outage is handled through your implementation. Lets assume you are ingesting from Kafka in to NiFi since you mentioned you use Kafka.  You would likely start that dataflow  with a ConsumeKafka processor.  Le's also assume you have a 3 node NiFi cluster and the Kafka topic you are consuming from has 12 partitions.   Since all nodes in your cluster will be executing the consumeKafka processor, each will be a consumer of that topic.  With a single concurrent task (default) configured on the ConsumeKafka, each of those 3  NiFi node's consumeKafka will be assigned 4 partitions each.  If you were to set the Concurrent tasks to 4, then you now have a total of 12 consumers (one for each Kafka partition).   Now lets assume one of your NiFi nodes goes down, Kafka will see a drop in the number of consumers from 12 to 8 and rebalance.  So consumption will continue with some of those consumers now being assigned multiple partitions until the down NiFi node comes back on line. That is just one scenario. In the case of using a NiFI Listen type processor (example: ListenTCP).  This starts a TCP socket listener on each node in the NiFi cluster on the same port.  In this case it would be the client or some external mechanism that would need to handle failover to a different node in the event a NiFi node goes down.  This is typically handled with an external load balancer which distributes data to all the NiFi nodes or switches to a different node when a node goes down. In the use case of something like ListSFTP, this processor would be configured to run on "primary node" only.  Zookeeper is responsible for electing a primary node and a cluster coordinator in a NiFi cluster.  NiFi processor components like ListSFTP are designed for primary node execution only and store state on the data listed in cluster state (within zookeeper).  If the current elected primary node goes down, another node in the NiFi cluster is elected the new primary node and the primary node only configured processors are started non that new node.  Last recorded state for that component reported to ZK by the previous primary node is pulled from ZK to the new primary node processor and it picks up listing from there.  Again you have redundancy. The only place in NiFi were you can have data delay, is when a NiFi node goes down while it still has active data in its connection queue(s).  Other nodes will not have access to that data on the other down node to take over work on it.  It will remain in that node's flowfile and content repositories until that node has been restored and can continue processing on that queued FlowFiles.  So it is important to protect those two NiFi repositories using RAID configured drives.  You can minimize impact in such cases through good flow design and use of back pressure to limit amount of FlowFiles that can queue on a NiFi node. Also keep in mind that while the flowfie and content repositories are tightly coupled to the flow.xml.gz, these items are not tightly coupled to a specific piece of hardware.  You can stand up an entirely new node for you cluster and move the flow.xml.gz, content repo and flowfile repo on to that node before starting it and that new node will continue processing the queued FlowFiles. Hope this helps, Matt ... View more

@LuxIsterica    "filename" is also another FlowFile attribute that is created by default on every FlowFile that is created in NiFi.  With some processors a filename can not be derived from or created based in the content that is received. ExecuteSQL (no inbound connection) and generateFlowFile processors are good examples here. In case like this, NiFi will just default to using the FlowFile's uuid as the filename also. Your statement " attribute "filename" that generated that executesql is "inherited" in all processors" is not accurate.  Processors do not inherit attributes.   A NiFi FlowFile exists of two parts: 1. FlowFile attributes/metadata -- These FlowFile attributes reside in heap memory and are also stored in the flowfile_repository.  It is these attributes which "flow" from one processor component to another in you dataflow you build on the canvas. Processors then have access to these FlowFile Attributes when they execute against a given FlowFile from the inbound connection.  Some processors as part of their execution will create additional attributes on a FlowFile before it is committed to the processor relationship that is assigned to a outbound connection. 2. FlowFile Content -- The actual content of a FlowFile is written to a claim in the content_repository.  It is only access as needed by a processor.  It does not reside in heap memory unless a processor needs to do so to perform its function. These FlowFile attributes can be changed as your FlowFile passes through different processors, but they belong to the FlowFile and not the processors at all.  So there is nothing you need to "preserve/save" in most cases. Hope this adds some clarity, Matt ... View more

@Paul Yang    I was in no way implying that you should have removed your NiFi nodes DN as a user identity in the NiFi-Registry.   The DN for every NiFi node must exist in NiFi-Registry and have been granted both proxy and read on "Can Manage Buckets" policies. NiFi nodes will regularly read the buckets in the NiFi-Registry to see if a newer version of your Version controlled PG exists (this is why read  on "Can manage buckets" is needed.).  The "?" is displayed when the NiFi nodes can not read the bucket. When a user in NiFi performs a version control action, the node will proxy the request n behalf of that user to the NiFi-Registry.  This is why all NiFi nodes must exist as users in NiFi-Registry and have the proxy policy granted to them. Only your initial admin user should have all policies except proxy.  That user should never be proxying anything. Thanks, Matt ... View more

@Cl0ck    Anytime the NiFi process fails to start or shuts back down the reason should be output in either the nifi-bootsrap.log (if startup failed during bootstrap) or the nifi-app.log (shutdown because of some exception during loading of the main NiFi child process). Start my looking at these logs for what the issue may be.   NiFi will fail to start if the service is already running. so execute ps -ef|grep nifi to see if there may already be a some NiFi process still running. There should have been no need to remove NiFi before going back and installing those additional services. Having multiple services running on the same host should not be an issue as CFM has each of these services by default starting up on different ports.  Running other services on the same host as NiFi is not recommended due to resource contention.  But ok for just testing or playing around.   Hope this helps, Matt ... View more

@pxm    NiFi sets not restriction on the data size that can be processed.   Ingested data becomes the content portion of a NiFi FlowFile and is written to the content repository.  The data is not read again unless a processor needs to read the content; otherwise, only the FlowFile attributes/metadata is passed from processor component to another component.  So you need to make sure you have sufficient storage space for the NIFi content_repository.  It is also strongly recommended that this dedicated storage separate from any other NiFi repository. Beyond that, any limitation here will be on network and disk IO.   Thanks, Matt ... View more

@Paul Yang    A couple observation based on provided information: 1.  You have need clientAuth enabled nifi.registry.security.needClientAuth=true  With this enabled in NiFi-Registry the only authentication method supported will be 2-way TLS.  If you want to support other authentication methods lik Spnego, LDAP, and/or kerberos, this property must be false.   2. Your users.xml (used for user authorization and not authentication) only contains one user.  I am assuming the user is :  CN=arch-fndtf04.beta1.fn, OU=NIFI And your authorizations.xml also show that this is the only user that has been authorized to a bunch of policies.  Your ldap user you want to authorize must exist in the users.xml file.  My guess here is that you did not set the "Initial admin" user in your authorizers.xml (which you did not share).  There are two possible providers in the authorizers were you can set the initial admin (file-user-group-provider - Only set here if your initial admin user is NOT coming from the ldap-user-group-provider.  file-access-provider - initial admin must be set here so initial admin policies are created for this user.   3. The users.xml and authorizations.xml files are only generated if they do not already exist.  If you go back and add an initial admin to your authorizers.xml, you will need to delete/rename the existing users.xml and authorizations.xml files o new ones can be generated in startup.   4. Since need clientAuth was set to true and UI clearly shows that user string:  CN=arch-fndtf04.beta1.fn, OU=NIFI successfully authenticated, your browser must have this client certificate loaded and presented to the server when you navigated to the URL for your NiFi-Registry.  I really see no reason why this certificate was loaded int to your browser.  NiFi-Registry, even when need clientAuth is false, will always try mutual TLS authentication first (becomes a WANT instead of REQUIRE when need clientAuth is false).  If no client certificate is provided in TLS handshake, next auth method tried is Spnego (if configured), and finally a configured login provider from the identity-providers.xml.   Hope this info helps with correcting your setup issues. Thanks, Matt   ... View more

@Paul Yang    NiFi stores information about the nodes that were connected to the cluster in its local state directory on each node.  Prior to securing your NiFi cluster, all your nodes were running unsecure on port 8080 and that information was retained in local state.  After securing your NiFi, you node were now identified using the same hostnames, but different secure port.  So Now your cluster is expecting to see 8080 and new secure port nodes in the cluster.   The easiest way to resolve this issue is to simply stop your NiFi nodes, remove the NiFi local state directory contents on each of your node, and then start your nodes.    Now if this is not a new installation and you are concerned about losing locally stored NiFi component state (like listFile stores state in what was already listed), then you can get around this issue by changing the configured cluster protocol port (found in the nifi.properties file) to a different unused port on your servers.  You will then be able to access the UI were you will see your new nodes plus the unexpected ones on the old ports which you can manually remove from the "cluster" UI accessible from the NiFi Global menu in the upper right corner.   This is a known issue which is being addressed.  But it only happens when you change ports (which commonly only happens when you are switching from unsecure to secure.   Hope this helps, Matt ... View more

@pauljoshiva    What this error is telling is that the flow.xml.gz on this node does not match the flow.xml.gz that is running on another node.  On startup of all your NiFi nodes an election takes place to determine which flow.xml.gz will be elected as the cluster flow.  So node 1 presents its flow (1 vote), then node 2 presents its flow (if it matches node 1's flow that flows vote increases to 2, if not it gets its own singular vote), this process repeats for all nodes joining the cluster until a flow.xml.gz is elected to be the cluster vote.  At that point in time, any nodes that have a flow.xml.gz that matched this elected cluster flow will throw the exception you  reported and shut back down. Since all nodes in your cluster must be running the exact same flow.xml.gz, you can copy the flow.xml.gz from one of the nodes that is up and joined in to the cluster to the node that threw the exception and restart it.  It should successfully join the cluster at on restart this time. Hope this helps, Matt ... View more

@Paul Yang    Ranger is not offered in CFM, but will become part of the platform in the future. The only authorization offering within NiFi and NiFi-Registry within CFM is the local file based authorizer. NiFi user and group authorization is controlled via the NiFi UI instead of through an external authorization provider like Ranger.  This same local file base authorization was also an option in HDF. https://docs.cloudera.com/cfm/1.0.1/securing-cfm/topics/cfm-enabling-tls.html You can configure NiFi to sync users and groups from LDAP also.  You can then through the NiFi UI assign authorization policies to these sync'd user and groups.   Thank you, Matt https://docs.cloudera.com/cfm/1.0.1/securing-cfm/topics/cfm-nifi-user-sync-ldap-properties.html ... View more