@zizo  The ConsumeKafka processor acts as a Kafka consumer group.  So makes sense that you set 30 concurrent tasks (1 per partition) assuming this is a single instance of NiFi.  If you had a 3 node NiFi cluster, you would set the concurrent tasks to 10 (10 x 3 nodes = 30 consumers).  What version of Kafka Server are you consuming from? MergeContent merging 100,000 FlowFiles (min). You did not share any component configurations here.  Is the MergeContent configured correctly to make sure each merge generated is 100,000 FlowFiles? Generally speaking, I would recommend using two MergeContent processors in series to reduce NiFi heap memory usage.  The more FlowFiles allocated to MergeContent bins, the more NiFi heap usage.  So first MergeContent merging say 20,000 min, followed by second MergeContent merging 5 min would achieve the same but with lower heap usage. Update Attribute does not touch the content of a FlowFile.  It simply updated metadata/attributes about a FlowFile.   PutSFTP throughput is for the most part dependent on the target SFTPP server and the network between NiFi and that SFP server.  Most SFTP servers only allow max 10 concurrent collections from the same client.  Did you configure this processor with 10 concurrent tasks?    Having a NiFi cluster would allow multiple NiFi nodes to send data concurrent to the SFTP server (10 concurrent tasks x 3). Are you saying the FlowFiles start queuing up at the putSFTP processor eventually leading to backpressure being applies all the way back through your dataflow until you reach the ConsumeKafka processor? Have you looked at CPU, disk I/O, network bandwidth and speed, NiFi heap usage? If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@plummcrazy  Not sure I understand the use case here.   You are looking for a way to pass an IP address and get the returned hostname and put it in a FlowFile attribute? The QueryDNS does not require any additional "special permissions" to use it. I have never tried using this processor myself. There are no other DNS specific processor designed to perform such task, you could need utilize the executeStreamCommand, ExecuteScript, or ExecuteGroovyScript processors to execute the system level command to return your reverse DNS lookup results. Execute based processor will require an additional layer of authorization since the processor is executed like every other processor as the NiFi service user (this means that ANY access the NiFi service user has will be accessible by the NiFi dataflow using these processors).  If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@luv4diamonds  NiFi will NOT form a cluster with a single instance Zookeeper.  Zookeeper must have quorum otherwise a cluster coordinator and primary node will not get elected for NiFi.  For Quorum you should have an odd number of ZK nodes with a minimum of 3. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@anony  You can not use NiFi Expression Language (NEL) in component property names. NEL can only be used in component property values and even then, only when the component property supports Expression language. With ExtractText, any dynamic properties added will create a new FlowFile Attribute on the outbound FlowFile with an attribute name matching the property name "${reasonId}" and the attribute set the the capture group string extracted from the source FlowFiles content. Without understanding yoru complete use case, it is difficult to offer alternative possible solutions. Why is it important to set a FlowFile property name to a reasonID?  FlowFile attribute property names only mean something to NiFi.  So if you were able to create dynamic property names, then how would you programmatically use them later in your dataflow as FlowFiles will have a variety of Attribute property names? If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt   ... View more

@databoi  I see from your images that you are using Apache NiFi 1.11.4 which is around the time that the Load Balanced connection capability was introduced.  There were many bugs subsequently identified in load balanced connection and addressed in future releases. I strongly encourage you to upgrade to the latest NiFi release and see if your issue persists.    If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@ushasri  NiFi does not provide a local provider for user authentication via username and password.  NiFI supports numerous authentication methods you can configure. - Authentication via a mutual TLS exchange:  Always enabled once NiFi is secured and the first method always attempted.  Mutual TLS is how NiFi nodes authenticate with one another, how Secured NiFi authenticates to other secured NiFis, how secured NiFi authenticates with secured NiFi-Registry, etc.... - Spnego: Kerberos authentication method that does not require users to provide a kerberos principal and password for authentication.   - Kerberos-provider: Requires user to provide username and password for authentication.  Successful authentication results in a bearer token issued to user and a corresponding NiFi side server token for the user. - Ldap-provider: Requires user to provide username and password for authentication.  Successful authentication results in a bearer token issued to user and a corresponding NiFi side server token for the user. - OpenID connect - SAML In a multi-node NiFi cluster, the authentication process and resulting bearer token is only valid for the node on which the authentication was verified.  Can't be used to authenticate with another node.  Issued bearer tokens have an expiration.  Default is 12 hours at which time the NiFi server removes the server side token.  This forces user to re-authenticate and obtain a new client bearer token.   Every action that a user performs via NiFi's UI must be both authenticated and authorized.  This mean that the user must authenticate via a certificate, bearer token, etc. with every request. The authenticated client user identity is then used to verify that the user/client is authorized to perform the requested operation or access the requested endpoint (such as loading the NiFi UI). Whenever a user authenticates within a NiFi instance, that NiFi instance will produce a log entry in the nifi-user.log with the authenticated user's user identity.  Subsequent to that is the requested endpoint that user is trying to access and result of that authorization. NOTE: Components added to the NiFi canvas by a user are not executed as that user. all components are executed by the NiFi service user.   Adding the username to a FlowFile attribute becomes challenging since the nothing in the NiFi canvas executes as that user.  Also running any kind of automation within the UI to make a rest-api call to NiFi would itself require user authentication and authorization. The question here is what are you trying to accomplish by extracting a username and adding it to a database? Guess you could create a dataflow that reads the nifi-user.log and extracts authentication related log lines, although, this would be very noisy since a single user who has obtained a bearer token for 12 hours may perform an unknown number of operations within the UI  or via rest-api calls each resulting in logging about authentication.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@davehkd  Thank you for the awesome feedback.  Glad i could help you. It would be great if you could go through this back and forth exchange and accept all the responses I provided that helped you.  I noticed you accepted your own response.  🙂 Matt ... View more

@niclyx  I think you are diverging form the original query asked in this community question which has an accepted solution. When it comes to a mutual TLS exchange/handshake, this is not something NiFi created or has made any modification to.  There are many different resources on the web to learn about MutualTLS and how it works.  NiFi only cares that the TLS handshake was successful and if no other method of authentication was setup, it received a clientAuth certificate from which it will use the full DN as the identity for the authenticated user. I recommend starting a new community questions if you have new questions. Matt ... View more

@Manika  The single-user-provider used by NiFi out of the box does not support multiple users.  It simply sets up a single user to access the secured NiFi suing the out of the box NiFi setup.  Only the very first time NiFi is launched with this setup will the generated username and password be output in the nifi-app.log.  You can use the following command to change that username and password: $ ./bin/nifi.sh set-single-user-credentials <username> <password> This single user username and password has full access to everything, so NiFi does not expose "policies" as setting up additional users is not possible.   NiFi provides no other provider for local users.  NiFi historically has left user management as an external dependency.  In order tp support multiple users with and various level of authorization, you will need to switch to using a different authorizer and authentication method. Switching to another authentication method will require you establish an external means for user authentication.  NiFi has numerous supported methods (User certificate, LDAP, Kerberos, etc.) https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication Once you have selected the method you will use to authenticate your users that is not the "single-user-provider", you need to change your authorizer from "single-user-authorizer" to "standard-managed-authorizer": https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#standardmanagedauthorizer   If you found that any of the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more