Member since
11-15-2016
1
Post
1
Kudos Received
0
Solutions
11-15-2016
02:46 PM
1 Kudo
I get an ClosedChannelException when I try
to read kafka topics from my storm spout. Storm java snipped: BrokerHosts hosts = new ZkHosts(zkHostString, "/brokers");
SpoutConfig spoutConfig = new SpoutConfig(hosts, "testTopic", "/myOffsets/testTopic", "myGroup");
spoutConfig.securityProtocol = "PLAINTEXTSASL";
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig); In the logs you can see, that no
GlobalPartitionInformation can be found. 2016-11-14 21:43:59.846 o.a.s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{topic=testTopic, partitionMap={0=null:-1, 1=null:-1}} In my opinion, that is because of the
information in zookeeper are stored in the endpoint instead of HOST and PORT
fields. [zk: zk1370:2181(CONNECTED) 10] get /brokers/ids/1001
{"jmx_port":-1,"timestamp":"1477657407466","endpoints":["SASL_PLAINTEXT://broker0190:6667","SASL_SSL://broker0190:6668"],"host":null,"version":3,"port":-1}
[zk: zk1370:2181(CONNECTED) 9] get /brokers/ids/1002
{"jmx_port":-1,"timestamp":"1477658692052","endpoints":["SASL_PLAINTEXT://broker0210:6667","SASL_SSL://broker0210:6668"],"host":null,"version":3,"port":-1}
[zk: zk1370:2181(CONNECTED) 12] get /brokers/topics/testTopic
{"version":1,"partitions":{"1":[1001,1002],"0":[1002,1001]}} The odd thing is, on a non-kerberos
cluster these values are not set to null but to the proper hostnames and port. See also this similar question: Getting Kafka Consumer offsets - getting null for both kafka-consumer-groups.sh and kafka-consumer-offset-checker.sh
Therefore my question: Why are the host and port values in a
kerberized cluster NULL and what is the best practice to use the endpoint
information for my kafka spout?
... View more
Labels: