@hackerway  I would not recommend using the embedded zookeeper as zookeeper quorum must be maintained in order for yoru NiFi cluster to remain stable.  Losing a NiFi node also takes down the embedded zookeeper running with that NiFi.   While I would recommend keeping your zookeeper quorum installed on its own servers separate from the NiFi service (NiFi can be a resource intensive application depending on dataflow designs and data volumes), you may be able to run yoru external ZK servers and NiFi servers on the same hardware. Sounds like you have established dataflows and resource usage is low on your servers, but that may change if you add to your existing dataflows, add new dataflows, and/or there is a change in data size/volume.  You'll want to be looking at all resource consumption over time before making a decision here: - CPU Load (core load average as well as peaks) - Memory utilization on yoru servers to make sure there is sufficient memory to support both ZK and NiFi. - Disk I/O NiFi can have a lot of disk I/O with it local repositories (content, flowfile, and provenance). So if Disk I/O is high adding ZK to the same host could impact throughput of your NiFi If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@hegdemahendra  SASL_SSL is Simple Authentication and Security Layer over a TLS encrypted connection. A TLS encrypted connection is going to require that the server (Kafka) has a server certificate.  There are two types of TLS connection (1-way and mutual). Without going very deep in to this process, i'll keep it very high level).  In a 1-way TLS connection only the server (Kafka in your case) presents information in the TLS exchange.  The client (NiFi Kafka processor in your case) does not present information back to the server. The client will verify if they trust the information provided from the server.  The servers serverAuth certificate will be signed by some certificate authority and the client will verify if that signing authority's trust chain is trusted and if if it is will accept the Server's information allowing the TLS connection to proceed.   Now it is possible for the client to choose to trust the server (unsecure) without verifying the trust chain.  This is unsafe, so NiFi and its components (Processors, controller services, reporting tasks, RPG, etc) will not allow that.  So the processor will require that a SSL Context Service is created.  You'll need to configure that SSL Context Service with a SSL truststore which contains the complete trust chain for the Kafka signed certificate.  No need for configuring the keystore properties. In a Mutual TLS connection, both sides (server and Client) will present information to each other to identify themselves in order to establish the encrypted TLS connection.  In a mutual TLS exchange, the Client must be capable via its truststore to trust the compete trust chain for the server's certificate issuer.  And the Server must be capable via its truststore to trust the complete trust chain for the clients certificate issuer. Since you are authenticating via SASL via username and password, you'll only need 1-way TLS encrypted connection. You can manually create a truststore.jks to use for this.  You should be able to use openssl to get the public keys needed to do this from yoru kafka target: openssl s_client -connect <kafka hostname>:<kafka port> -showcerts The output from this should contain all the public keys in the trust chain. Each certificate will look something like this: -----BEGIN CERTIFICATE----- MIIFYjCCBE......yilrbCgj8= -----END CERTIFICATE----- The "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" are part of the certificate. You can save each certificate as a "key<num>.pem" file and the import them in to your truststore you when then use in your SSLContextService. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@KleytonMayer  Best to provide the version of NiFi you are using along with the specific ConsumeKafka/ConsumeKafkaRecord processor you are using along with its configuration. I'd expect your ConsumeKafka to split out one FlowFile per consumed Kafka record unless you have changed setting defaults or you are using a ConsumeKafkaRecord processor. If you don't need to split your FlowFile into different FlowFiles for processing, I'd recommend you look into using the various "record" based processors NiFi offers.  Working with larger multi-record FlowFiles is more efficient and uses less resources. The output you shared looks like it may be a single complete JSON per line.  If so, you could simply use the SplitText processor with a line Split Count of 1. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@SandyClouds    Can you share the contents of your authorizers.xml file? This seems odd to me: "'CN=admin, OU=NiFi'" you show having both double quotes and single quotes around your user's DN in the users.xml file. Did you put single quotes around the DN in the authorizer's file-access-policy-provider and file-user-group-provider?  If so remove those single quotes, users.xml, and authorizations.xml, then restart your NiFi-Registry again. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt     ... View more