Error Kafka

[#1]

Hey, folks,

I have an error when executing kafka when executing a message for the topic.

kafka-console-producer.sh --broker-list w01.s03.hortonweb.com:6667 -topic TestNYC

error.

WARN [Producer clientId=console-producer] Bootstrap broker w01.s03.hortonweb.com:6667 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

I have 3 Brokers, which are working and is configured according to the parameters. just a topic that I just realized.

I have 2 network cards one internal and external to netstat I see that port 6667 is listening to the internal. but still executing the command with the internal IP

kafka-console-producer.sh --broker-list 192.168.0.9:6667 -topic TestNYC

I still get the same mistake.

Someone could help me.

Greetings

[#2]

@Peruvian81 

Hi,

If you connect to the zookeeper cli using:

ZK_HOME/zookeeper-client/bin/zkCli.sh -server <zkHost>:<zkPort>

Then you can run:  get /brokers/ids/<brokerID> and check in the "endpoints" where the kafka brokers are listening.

Then try using that security-protocol ip:port to connect to the brokers.

Also, make sure that the topic has all their replicas in sync by running "describe" command line.

Let us know how it goes.

[#3]

Hello @ManuelCalvo 

When you execute the command

[zookeeper@m01 bin]$ ./zkCli.sh -server m01.s02.hortonweb.com:2181 get /brokers/ids
Connecting to m01.s02.hortonweb.com:2181
2019-09-25 16:22:54,331 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-78--1, built on 12/06/2018 12:30 GMT
2019-09-25 16:22:54,333 - INFO [main:Environment@100] - Client environment:host.name=m01.s02.hortonweb.com
2019-09-25 16:22:54,333 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_112
2019-09-25 16:22:54,335 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-09-25 16:22:54,335 - INFO [main:Environment@100] - Client environment:java.home=/usr/jdk64/jdk1.8.0_112/jre
2019-09-25 16:22:54,335 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/hdp/current/zookeeper-client/bin/../build/classes:/usr/hdp/current/zookeeper-client/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/
current/zookeeper-client/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/netty-3.10.5.Final.jar:/usr/hdp/current/zookeeper-client/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeeper-client/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zookeeper-client/
bin/../zookeeper-3.4.6.3.1.0.0-78.jar:/usr/hdp/current/zookeeper-client/bin/../src/java/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../conf::/usr/share/zookeeper/*
2019-09-25 16:22:54,335 - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:os.version=3.10.0-957.12.1.el7.x86_64
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:user.name=zookeeper
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:user.home=/home/zookeeper
2019-09-25 16:22:54,336 - INFO [main:Environment@100] - Client environment:user.dir=/usr/hdp/3.1.0.0-78/zookeeper/bin
2019-09-25 16:22:54,337 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=m01.s02.hortonweb.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@67424e82
2019-09-25 16:22:54,367 - WARN [main-SendThread(m01.s02.hortonweb.com:2181):ZooKeeperSaslClient$ClientCallbackHandler@496] - Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make
sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine
if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <ke
ytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's cl
ock is in sync with this host's clock.
2019-09-25 16:22:54,369 - WARN [main-SendThread(m01.s02.hortonweb.com:2181):ClientCnxn$SendThread@1001] - SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper
server allows it.
2019-09-25 16:22:54,371 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server m01.s02.hortonweb.com/192.168.0.2:2181

WATCHER::

WatchedEvent state:AuthFailed type:None path:null
2019-09-25 16:22:54,443 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):ClientCnxn$SendThread@864] - Socket connection established, initiating session, client: /192.168.0.2:57438, server: m01.s02.hortonweb.com/192.168.0.2:2181
2019-09-25 16:22:54,571 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):ClientCnxn$SendThread@1279] - Session establishment complete on server m01.s02.hortonweb.com/192.168.0.2:2181, sessionid = 0x16ccd8510b02378, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
null
cZxid = 0x239
ctime = Tue May 07 10:07:46 CEST 2019
mZxid = 0x239
mtime = Tue May 07 10:07:46 CEST 2019
pZxid = 0x6b559
cversion = 103
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 3

I'm a little new to this subject. But I am impressed that by having 2 network cards my requests are going off without a specific direction. I have one for my LAN and one for WAN. both in Broker and Zookeeper.

Greetings

[#4]

@Peruvian81 

Are you using kerberos? If yes, make sure you have a valid ticket in order to avoid below exception:

2019-09-25 16:22:54,367 - WARN [main-SendThread(m01.s02.hortonweb.com:2181):ZooKeeperSaslClient$ClientCallbackHandler@496] - Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make
sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine
if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <ke
ytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's cl
ock is in sync with this host's clock.

From the command line, please add the broker id: get /brokers/ids/<brokerID>

Example:

ZK_HOME/zookeeper-client/bin/zkCli.sh -server host:2181 get /brokers/ids/1001

If you don't know your current ids, you can get them by using:

ZK_HOME/zookeeper-client/bin/zkCli.sh -server host:2181 ls /brokers/ids

[#5]

Hello @ManuelCalvo 

Generate a ticket with the kerberos admin and execute the command. But I don't see any error... excuse me I'm a little new to this topic. I attach the result of the command.

[root@m01 bin]# ./zkCli.sh -server m01.s02.hortonweb.com:2181 get /brokers/ids/1001
Connecting to m01.s02.hortonweb.com:2181
2019-09-26 12:09:27,940 - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-78--1, built on 12/06/2018 12:30 GMT
2019-09-26 12:09:27,942 - INFO [main:Environment@100] - Client environment:host.name=m01.s02.hortonweb.com
2019-09-26 12:09:27,942 - INFO [main:Environment@100] - Client environment:java.version=1.8.0_112
2019-09-26 12:09:27,944 - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-09-26 12:09:27,944 - INFO [main:Environment@100] - Client environment:java.home=/usr/jdk64/jdk1.8.0_112/jre
2019-09-26 12:09:27,944 - INFO [main:Environment@100] - Client environment:java.class.path=/usr/hdp/current/zookeeper-client/bin/../build/classes:/usr/hdp/current/zookeeper-client/bin/../build/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../lib/slf4j-log4j12-1.6.1.jar:/u
sr/hdp/current/zookeeper-client/bin/../lib/slf4j-api-1.6.1.jar:/usr/hdp/current/zookeeper-client/bin/../lib/netty-3.10.5.Final.jar:/usr/hdp/current/zookeeper-client/bin/../lib/log4j-1.2.16.jar:/usr/hdp/current/zookeeper-client/bin/../lib/jline-0.9.94.jar:/usr/hdp/current/zoo
keeper-client/bin/../zookeeper-3.4.6.3.1.0.0-78.jar:/usr/hdp/current/zookeeper-client/bin/../src/java/lib/*.jar:/usr/hdp/current/zookeeper-client/bin/../conf::/usr/share/zookeeper/*
2019-09-26 12:09:27,944 - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-09-26 12:09:27,944 - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-09-26 12:09:27,945 - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-09-26 12:09:27,945 - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-09-26 12:09:27,945 - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-09-26 12:09:27,945 - INFO [main:Environment@100] - Client environment:os.version=3.10.0-957.12.1.el7.x86_64
2019-09-26 12:09:27,945 - INFO [main:Environment@100] - Client environment:user.name=root
2019-09-26 12:09:27,945 - INFO [main:Environment@100] - Client environment:user.home=/root
2019-09-26 12:09:27,945 - INFO [main:Environment@100] - Client environment:user.dir=/usr/hdp/3.1.0.0-78/zookeeper/bin
2019-09-26 12:09:27,947 - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=m01.s02.hortonweb.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@67424e82
2019-09-26 12:09:28,051 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):Login@294] - successfully logged in.
2019-09-26 12:09:28,052 - INFO [Thread-0:Login$1@127] - TGT refresh thread started.
2019-09-26 12:09:28,056 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):ZooKeeperSaslClient$1@289] - Client will use GSSAPI as SASL mechanism.
2019-09-26 12:09:28,067 - INFO [Thread-0:Login@302] - TGT valid starting at: Thu Sep 26 09:16:58 CEST 2019
2019-09-26 12:09:28,067 - INFO [Thread-0:Login@303] - TGT expires: Thu Sep 26 19:16:58 CEST 2019
2019-09-26 12:09:28,067 - INFO [Thread-0:Login$1@181] - TGT refresh sleeping until: Thu Sep 26 17:26:26 CEST 2019
2019-09-26 12:09:28,104 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server m01.s02.hortonweb.com/192.168.0.2:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-09-26 12:09:28,160 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):ClientCnxn$SendThread@864] - Socket connection established, initiating session, client: /192.168.0.2:59854, server: m01.s02.hortonweb.com/192.168.0.2:2181
2019-09-26 12:09:28,317 - INFO [main-SendThread(m01.s02.hortonweb.com:2181):ClientCnxn$SendThread@1279] - Session establishment complete on server m01.s02.hortonweb.com/192.168.0.2:2181, sessionid = 0x16ccd8510b02493, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

WATCHER::

WatchedEvent state:SaslAuthenticated type:None path:null
{"listener_security_protocol_map":{"SASL_PLAINTEXT":"SASL_PLAINTEXT"},"endpoints":["SASL_PLAINTEXT://w01.s03.hortonweb.com:6667"],"jmx_port":-1,"host":null,"timestamp":"1569423123514","port":-1,"version":4}
cZxid = 0x6c420
ctime = Wed Sep 25 16:52:03 CEST 2019
mZxid = 0x6c420
mtime = Wed Sep 25 16:52:03 CEST 2019
pZxid = 0x6c420
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16ccd8510b0238e
dataLength = 205
numChildren = 0
[root@m01 bin]#

I execute these commands

./kafka-topics.sh --create --zookeeper m01.s02.hortonweb.com:2181 --replication-factor 3 (i have 3 Brokers)--partitions 1 --topic PruebaKafka
Created topic "PruebaKafka".

./kafka-console-producer.sh --broker-list w01.s03.hortonweb.com:6667 --topic PruebaKafka

>Helpme

[2019-09-26 12:22:22,059] WARN [Producer clientId=console-producer] Bootstrap broker w01.s03.hortonweb.com.com:6667 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2019-09-26 12:22:22,165] WARN [Producer clientId=console-producer] Bootstrap broker w01.s03.hortonweb.com.com:6667 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2019-09-26 12:22:22,270] WARN [Producer clientId=console-producer] Bootstrap broker w01.s03.hortonweb.com.com:6667 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

Greetings

[#6]

Hello @ManuelCalvo 

That's right. We're using Kerberos. Generate a ticket as admin and execute the command indicated and consulted a Broker.

I attach a URL with the report that comes out.

https://ibb.co/Bg1jfQx

I also indicate the commands that we executed when we created the topic and the producer.

./kafka-topics.sh --create --zookeeper m01.s02.hortonweb.com:2181 --replication-factor 3 --partitions 1 --topic PruebaKafka (I Have 3 Brokers)
Created topic "PruebaKafka".

./kafka-console-producer.sh --broker-list m01.s02.hortonweb.com:6667 --topic PruebaKafka
>Prueba mensaje

[2019-09-26 12:22:22,059] WARN [Producer clientId=console-producer] Bootstrap broker w01.s03.hortonweb.com.com:6667 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

Thank you

[#7]

@Peruvian81 

According to the output, the broker is listening on SASL_PLAINTEXT (kerberos) and host w01.s03.hortonweb.com. We have to specify the connection type we are doing from our clients to Kafka, by default the connection is PLAINTEXT, depending on the Kafka version in use, you should try the following:

1. Get a valid Kerberos token "kinit -kt <keytab> <principal>"

2. Execute the command below for Kafka version until 1.0.0

./kafka-console-producer.sh --broker-list w01.s03.hortonweb.com:6667 --topic PruebaKafka --security-protocol SASL_PLAINTEXT

Kafka 2.0 onwards:

./kafka-console-producer.sh --broker-list w01.s03.hortonweb.com:6667 --topic PruebaKafka --producer-property security.protocol=SASL_PLAINTEXT

Test and let us know.

[#8]

@ManuelCalvo 

[root@w01 bin]# ./kafka-console-producer.sh --broker-list w01.s03.hortonweb.com:6667 --topic PruebaNYC --producer-property security.protocol=SASL_PLAINTEXT
>Hello
>Helo

Thank you very much it worked for me, I don't know if it is to ask a lot and what would be the command for the consumer.

Thank you

[#9 · Accepted solution]

@Peruvian81 

You can try below command for the consumer:

./kafka-console-consumer.sh --bootstrap-server w01.s03.hortonweb.com:6667 --topic PruebaNYC --consumer-property security.protocol=SASL_PLAINTEXT --from-beginning

If that solves your issue, kindly put this thread as solved. 

Thanks.