How to connect to Kafka from a Windows remote machine

[#1]

This is a step by step guide to test Kafka clients from a Windows machine that connects to an HDF/HDP environment.

We start with the review of the current Kafka broker listeners. In this case, we will cover the following:

• SASL_PLAINTEXT > Kerberized environments
• PLAINTEXT > Plain connections

This can be done by using the Ambari console > Kafka > configs > Kafka Broker. After that, search for listeners and make sure either one or both protocols are enabled.

PLAINTEXT security protocol

1. Go to your Windows machine and download the apache Kafka software.
   1. It is recommended to download the same version that it's running in your HDP/HDF cluster. Select the "Scala 2.12" link to avoid exceptions while running the Kafka clients.
   2. Extract the content of this folder in a preferred location in the Windows host.
2. While connecting to Kafka through PLAINTEXT listener, Kafka does not have a way to identify you as a user. Hence, add Kafka ACLs and give permissions to ANONYMOUS users. To achieve this run the following command as a Kafka user in one of the Kafka brokers:

   /usr/hd<p/f>/current/kafka-broker/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=<zkHost>:<zkPort> --add --allow-principal User:ANONYMOUS --operation All --topic=* --group=* --cluster

    The above command will give all permissions to the anonymous user in Kafka, change the topic and group to specific ones if required.
3. In a Kafka host, create a new test topic or use an existing one. To create a new topic, run the following command with the Kafka user:

   kafka-topics --create --topic <topicName> --partitions <N of partitions> --replication-factor <N of replicas> --zookeeper <zkHost>:<zkPort>

4. When adding anonymous user permissions, go to our Windows Machine and navigate to the following Kafka folder: 
   Note: This step assumes that we already have connectivity to the brokers and the firewall and DNS (if any) are configured properly.
   

   C:\<preferred location>\kafka_<version>\bin\windows

   
   
   1. In this folder, there is a list of .bat files, similar to the ones in Linux hosts with .sh extension. In order to run .bat producer, use the following command:

      C:\<preferred location>\kafka_<version>\bin\windows\bin\kafka-console-producer.bat --broker-list <brokerHost>:<brokerPort> --topic <topicName>

   2. To run a consumer, please run the following command:

      C:\<preferred location>\kafka_<version>\bin\windows\bin\kafka-console-consumer.bat --bootstrap-server <brokerHost>:<brokerPort> --topic <topicName> --from-beginning

Run the clients using Kerberos (SASL_PLAINTEXT)

To run the clients using Kerberos (SASL_PLAINTEXT), first ensure that Kerberos is configured properly in the environment. Once you get valid tickets, do the following to connect with the Kafka clients:

1.  If using Kafka Ranger plugin, go to Ranger Admin UI -> Kafka and add a new policy for the user that is used to connect from Windows host pointing to the topic/s that needs access.
2. After the Ranger policies are configured, then go to the Windows Host and configure the Kerberos details for the Kafka client connection. To achieve this, do the following:
   1. Create a file with extension .conf and add the following content:

          KafkaClient {
          com.sun.security.auth.module.Krb5LoginModule required
          useKeyTab=true
          useTicketCache=false
          serviceName="kafka";
          keyTab="/path_to_file/file.keytab"
          principal="[email protected]";
          };
          Client {
          com.sun.security.auth.module.Krb5LoginModule required
          useKeyTab=true
          keyTab="/path_to_file/file.keytab"
          storeKey=true
          useTicketCache=false
          serviceName="zookeeper"
          principal="[email protected]";
          };

      • client: is used to connecting to the Zookeeper and KafkaClient is to connect to the Kafka Brokers.
      • principal: is the user that will be used to connect from Windows to the Kafka Brokers (the same user that we add grants in Ranger UI)
      • keyTab: is the keytab file that contains the principal specified in "principal". 
   2. With that file created, open a Windows Command Prompt and execute the following command before running any command line:

      set KAFKA_OPTS="-Djava.security.auth.login.config=/path_to_conf_file/file.conf"

      
      That command will pass the keytab/principal to the Kafka client.
   3. In the same command prompt, run a Kafka Producer/Consumer using the following commands for Kafka versions <= 1.0:

      C:\<preferred location>\kafka_<version>\bin\windows\bin\kafka-console-producer.bat --broker-list <brokerHost>:<brokerPort> --topic <topicName> --security-protocol SASL_PLAINTEXT

      
      For the consumer, use the following command line:

      C:\<preferred location>\kafka_<version>\bin\windows\bin\kafka-console-consumer.bat --bootstrap-server <brokerHost>:<brokerPort> --topic <topicName> --from-beginning --security-protocol SASL_PLAINTEXT

      
      For Kafka versions > 1.0, use the following producer/consumer command line:

      C:\<preferred location>\kafka_<version>\bin\windows\bin\kafka-console-producer.bat --broker-list <brokerHost>:<brokerPort> --topic <topicName> --producer-property security.protocol=SASL_PLAINTEXT

      
      For Kafka consumer > 1.0, use the following command line

      C:\<preferred location>\kafka_<version>\bin\windows\bin\kafka-console-consumer.bat --bootstrap-server <brokerHost>:<brokerPort> --topic <topicName> --from-beginning 
      --consumer-property security.protocol=SASL_PLAINTEXT