@LuxIsterica    "filename" is also another FlowFile attribute that is created by default on every FlowFile that is created in NiFi.  With some processors a filename can not be derived from or created based in the content that is received. ExecuteSQL (no inbound connection) and generateFlowFile processors are good examples here. In case like this, NiFi will just default to using the FlowFile's uuid as the filename also. Your statement "attribute "filename" that generated that executesql is "inherited" in all processors" is not accurate.  Processors do not inherit attributes.   A NiFi FlowFile exists of two parts: 1. FlowFile attributes/metadata -- These FlowFile attributes reside in heap memory and are also stored in the flowfile_repository.  It is these attributes which "flow" from one processor component to another in you dataflow you build on the canvas. Processors then have access to these FlowFile Attributes when they execute against a given FlowFile from the inbound connection.  Some processors as part of their execution will create additional attributes on a FlowFile before it is committed to the processor relationship that is assigned to a outbound connection. 2. FlowFile Content -- The actual content of a FlowFile is written to a claim in the content_repository.  It is only access as needed by a processor.  It does not reside in heap memory unless a processor needs to do so to perform its function. These FlowFile attributes can be changed as your FlowFile passes through different processors, but they belong to the FlowFile and not the processors at all.  So there is nothing you need to "preserve/save" in most cases. Hope this adds some clarity, Matt ... View more

@Paul Yang    I was in no way implying that you should have removed your NiFi nodes DN as a user identity in the NiFi-Registry.   The DN for every NiFi node must exist in NiFi-Registry and have been granted both proxy and read on "Can Manage Buckets" policies. NiFi nodes will regularly read the buckets in the NiFi-Registry to see if a newer version of your Version controlled PG exists (this is why read  on "Can manage buckets" is needed.).  The "?" is displayed when the NiFi nodes can not read the bucket. When a user in NiFi performs a version control action, the node will proxy the request n behalf of that user to the NiFi-Registry.  This is why all NiFi nodes must exist as users in NiFi-Registry and have the proxy policy granted to them. Only your initial admin user should have all policies except proxy.  That user should never be proxying anything. Thanks, Matt ... View more

@Cl0ck    Anytime the NiFi process fails to start or shuts back down the reason should be output in either the nifi-bootsrap.log (if startup failed during bootstrap) or the nifi-app.log (shutdown because of some exception during loading of the main NiFi child process). Start my looking at these logs for what the issue may be.   NiFi will fail to start if the service is already running. so execute ps -ef|grep nifi to see if there may already be a some NiFi process still running. There should have been no need to remove NiFi before going back and installing those additional services. Having multiple services running on the same host should not be an issue as CFM has each of these services by default starting up on different ports.  Running other services on the same host as NiFi is not recommended due to resource contention.  But ok for just testing or playing around.   Hope this helps, Matt ... View more

@pxm    NiFi sets not restriction on the data size that can be processed.   Ingested data becomes the content portion of a NiFi FlowFile and is written to the content repository.  The data is not read again unless a processor needs to read the content; otherwise, only the FlowFile attributes/metadata is passed from processor component to another component.  So you need to make sure you have sufficient storage space for the NIFi content_repository.  It is also strongly recommended that this dedicated storage separate from any other NiFi repository. Beyond that, any limitation here will be on network and disk IO.   Thanks, Matt ... View more

@Paul Yang    A couple observation based on provided information: 1.  You have need clientAuth enabled nifi.registry.security.needClientAuth=true  With this enabled in NiFi-Registry the only authentication method supported will be 2-way TLS.  If you want to support other authentication methods lik Spnego, LDAP, and/or kerberos, this property must be false.   2. Your users.xml (used for user authorization and not authentication) only contains one user.  I am assuming the user is :  CN=arch-fndtf04.beta1.fn, OU=NIFI And your authorizations.xml also show that this is the only user that has been authorized to a bunch of policies.  Your ldap user you want to authorize must exist in the users.xml file.  My guess here is that you did not set the "Initial admin" user in your authorizers.xml (which you did not share).  There are two possible providers in the authorizers were you can set the initial admin (file-user-group-provider - Only set here if your initial admin user is NOT coming from the ldap-user-group-provider.  file-access-provider - initial admin must be set here so initial admin policies are created for this user.   3. The users.xml and authorizations.xml files are only generated if they do not already exist.  If you go back and add an initial admin to your authorizers.xml, you will need to delete/rename the existing users.xml and authorizations.xml files o new ones can be generated in startup.   4. Since need clientAuth was set to true and UI clearly shows that user string:  CN=arch-fndtf04.beta1.fn, OU=NIFI successfully authenticated, your browser must have this client certificate loaded and presented to the server when you navigated to the URL for your NiFi-Registry.  I really see no reason why this certificate was loaded int to your browser.  NiFi-Registry, even when need clientAuth is false, will always try mutual TLS authentication first (becomes a WANT instead of REQUIRE when need clientAuth is false).  If no client certificate is provided in TLS handshake, next auth method tried is Spnego (if configured), and finally a configured login provider from the identity-providers.xml.   Hope this info helps with correcting your setup issues. Thanks, Matt   ... View more

@Paul Yang    NiFi stores information about the nodes that were connected to the cluster in its local state directory on each node.  Prior to securing your NiFi cluster, all your nodes were running unsecure on port 8080 and that information was retained in local state.  After securing your NiFi, you node were now identified using the same hostnames, but different secure port.  So Now your cluster is expecting to see 8080 and new secure port nodes in the cluster.   The easiest way to resolve this issue is to simply stop your NiFi nodes, remove the NiFi local state directory contents on each of your node, and then start your nodes.    Now if this is not a new installation and you are concerned about losing locally stored NiFi component state (like listFile stores state in what was already listed), then you can get around this issue by changing the configured cluster protocol port (found in the nifi.properties file) to a different unused port on your servers.  You will then be able to access the UI were you will see your new nodes plus the unexpected ones on the old ports which you can manually remove from the "cluster" UI accessible from the NiFi Global menu in the upper right corner.   This is a known issue which is being addressed.  But it only happens when you change ports (which commonly only happens when you are switching from unsecure to secure.   Hope this helps, Matt ... View more

@pauljoshiva    What this error is telling is that the flow.xml.gz on this node does not match the flow.xml.gz that is running on another node.  On startup of all your NiFi nodes an election takes place to determine which flow.xml.gz will be elected as the cluster flow.  So node 1 presents its flow (1 vote), then node 2 presents its flow (if it matches node 1's flow that flows vote increases to 2, if not it gets its own singular vote), this process repeats for all nodes joining the cluster until a flow.xml.gz is elected to be the cluster vote.  At that point in time, any nodes that have a flow.xml.gz that matched this elected cluster flow will throw the exception you  reported and shut back down. Since all nodes in your cluster must be running the exact same flow.xml.gz, you can copy the flow.xml.gz from one of the nodes that is up and joined in to the cluster to the node that threw the exception and restart it.  It should successfully join the cluster at on restart this time. Hope this helps, Matt ... View more