How to pass Credentials to the KafkaRecordSink

[#1]

I am trying to utilize the KafkaRecordSink_2_6 as a destination for a couple of my reporting tasks.  However, I do not see a way to pass in the Security Mechanism or the credentials as I do on Kafka processors.

I'm guessing this has a lot to do with my unfamiliarity with Kafka authentication.  I do see that there is support for a Kerberos configuration, however we are not using that mechanism for our other connections.

I'm hoping that there are some "magic" / undocumented properties on this controller which will allow it to be used in our environment.

Any recommendations?  I'd really like to get some of the system events and bulletins posted into a topic so I can process them.  (Without the overhead of saving to the database...then using Kafka Connect or another processor to pull them back out into a queue...it's doable, but seems clunky.)

[#2]

@kellerj 

You should use the following Kafka settings in this nifi processor:

1) Topic name 

2) Broker name and it's port which you can check from the Kafka configuration 

3) Kafka security protocol. You should check it from the Kafka side which protocol you are using whether it's SASL_PLAINTEXT or SASL_SSL

4) Keytab and principal which you want to use and having permission to produce data to Kafka topic 

5) If you are using SASL_SSL protocol, then you will need to use Kafka SSL details.

Note 1: For the 4th and 5th number points you can add control services in your processors like KeytabCredentialsService & StandardRestrictedSSLContextService

Note 2: Keytab and SSL files (keystore & truststore) should be present on all nifi nodes and they should be accessible by the user. If ssl.client.auth=required is enabled in kafka >> conf, then you should use both keystore & truststore. If it's disabled, then use truststore only.

If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

[#3]

Thanks for the info.  I'm not actually sure what to do with the KeyTabCredentialsService as we are not using Kerberos on our NiFi nodes at present.  So, without any configuration in /etc/krb5.conf or setting up a KDC server, is there a way to use that controller service?

[#4]

@kellerj 

If this is not kerberized cluster then there is no need to use the KeyTabCredentialsService controller service. 

As per your query initially, we have provided all security details that you can use from the Nifi processor to connect to Kafka. But here in this case you should know what security parameters you are using for Kafka and use those details as explained above in the Nifi processor.

Thank you.

[#5]

@kellerj Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks