Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron storm topologies - wrong kafka config

Metron storm topologies - wrong kafka config

New Contributor

I'm having an issue where all Storm topologies for Metron have bad Kafka config (port and protocol) for spouts and bolts and can't seem to track it down. Any help is appreciated.

  • Ambari Version
  • HDP-
  • Mpack version: metron-ambari.mpack-

In Ambari, I have the following config for Kafka

  • listeners = SASL_PLAINTEXT://localhost:6667,PLAINTEXT://localhost:6668
  • port = 6667

From Ambari (Metron Indexing Start - but happening on all topologies)

Running: /usr/jdk64/jdk1.8.0_112/bin/java -Dstorm.options= -Dstorm.home=/usr/hdp/ -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/ -Dstorm.jar=/tmp/7d39e2f050b311e892b3a0369fe7cbc4.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.flux.Flux --remote /usr/hcp/ --filter /usr/hcp/

In /usr/hcp/

##### Kafka #####,,,,

From worker.log (kafka on is stopped - assuming that's why it's not in bootstrap.servers?)

2018-04-29 18:54:43.340 o.a.k.c.c.ConsumerConfig Thread-15-kafkaSpout-executor[5 5] [INFO] ConsumerConfig values:
        metric.reporters = [] = 300000
        partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor] = 50
        sasl.kerberos.ticket.renew.window.factor = 0.8
        max.partition.fetch.bytes = 1048576
        bootstrap.servers = [,]
        ssl.keystore.type = JKS = false
        sasl.mechanism = GSSAPI
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.keystore.location = null
        ssl.cipher.suites = null
        security.protocol = PLAINTEXTSASL
        ssl.keymanager.algorithm = SunX509 = 30000
        auto.offset.reset = latest
2018-04-29 18:54:44.075 o.a.k.c.n.Selector Thread-15-kafkaSpout-executor[5 5] [WARN] Unexpected error from; closing connection
org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected handshake request with client mechanism GSSAPI, enabled mechanisms are [PLAIN, GSSAPI]

I did some more digging but I'm grasping at straws. It looks like from the properties file should be piped into the yaml files. I see the discrepancy with pcap but haven't gotten that far in testing yet...

[root@metronhost workers-artifacts]# grep -r ""  /usr/hcp/*
/usr/hcp/                  - "${}"
/usr/hcp/                  - "${}"
/usr/hcp/                  - "${}"
/usr/hcp/                  - "${}"
/usr/hcp/                    - "${}"
/usr/hcp/                    - "${}"
/usr/hcp/                    - "${}"
/usr/hcp/                    - "${}"
/usr/hcp/                    - "${}"
/usr/hcp/                    - "${}"
/usr/hcp/            - "${}"

But...the kafka brokers and port are pulled from ZooKeeper from /brokers/ids/ ( if it's not found in ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG. I might be on a fishing expedition but this is the only place port = 6668 beisides in the PLAINTEXT endpoint.




Don't have an account?
Coming from Hortonworks? Activate your account here