Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron storm topologies - wrong kafka config

Metron storm topologies - wrong kafka config

New Contributor

I'm having an issue where all Storm topologies for Metron have bad Kafka config (port and protocol) for spouts and bolts and can't seem to track it down. Any help is appreciated.

  • Ambari Version 2.6.1.5
  • HDP-2.6.4.0
  • Mpack version: metron-ambari.mpack-0.4.3.0

In Ambari, I have the following config for Kafka

  • security.inter.broker.protocol = SASL_PLAINTEXT
  • listeners = SASL_PLAINTEXT://localhost:6667,PLAINTEXT://localhost:6668
  • port = 6667

From Ambari (Metron Indexing Start - but happening on all topologies)

Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/hdp/2.6.4.0-91/storm -Dstorm.log.dir=/var/log/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/hdp/2.6.4.0-91/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.4.0-91/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.4.0-91/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.4.0-91/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.4.0-91/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.4.0-91/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.4.0-91/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.4.0-91/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.4.0-91/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.4.0-91/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.4.0-91/storm/lib/storm-core-1.1.0.2.6.4.0-91.jar:/usr/hdp/2.6.4.0-91/storm/lib/storm-rename-hack-1.1.0.2.6.4.0-91.jar:/usr/hdp/2.6.4.0-91/storm/lib/zookeeper.jar:/usr/hdp/2.6.4.0-91/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jackson-core-asl-1.9.13.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/HikariCP-2.5.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jackson-databind-2.6.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/ST4-4.0.4.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/activation-1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jackson-xc-1.9.13.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/ant-1.9.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-aws-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/ant-launcher-1.9.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/javassist-3.18.1-GA.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/antlr-2.7.7.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-client-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/antlr-runtime-3.4.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jdo-api-3.0.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/aopalliance-1.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/groovy-all-2.4.11.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jpam-1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/apache-log4j-extras-1.2.17.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jline-0.9.94.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/asm-3.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/javax.inject-1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/asm-commons-3.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/javax.jdo-3.2.0-m3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/asm-tree-3.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jaxb-api-2.2.2.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/avro-1.7.4.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-common-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-hdfs-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/gson-2.2.4.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jta-1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hbase-client-1.1.2.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/azure-storage-5.4.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hbase-common-1.1.2.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/bonecp-0.8.0.RELEASE.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hbase-protocol-1.1.2.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-beanutils-1.7.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/guava-16.0.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/oro-2.0.8.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-ant-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-cli-1.3.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-cli-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-codec-1.6.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/guice-3.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/joni-2.1.2.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-collections-3.2.2.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-common-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-compress-1.4.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/guice-servlet-3.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/xz-1.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-configuration-1.6.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jcip-annotations-1.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-dbcp-1.4.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-exec-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-digester-1.8.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-serde-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-httpclient-3.0.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jcodings-1.0.8.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-io-2.5.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jettison-1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-lang-2.5.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-service-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-lang3-3.4.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-shims-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-logging-1.2.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-math3-3.1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jetty-6.1.26.hwx.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-net-3.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/httpcore-4.4.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/commons-pool-1.5.4.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/httpclient-4.3.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/curator-client-2.12.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/ivy-2.4.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/curator-framework-2.12.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jackson-annotations-2.2.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/curator-recipes-2.12.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-annotations-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/datanucleus-api-jdo-4.2.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/datanucleus-core-4.1.6.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jackson-core-2.6.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/datanucleus-rdbms-4.1.7.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jetty-all-7.6.0.v20120127.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/derby-10.10.2.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-auth-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/findbugs-annotations-1.3.9-1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jms-1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/fst-2.24.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/geronimo-jta_1.1_spec-1.1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/geronimo-annotation_1.0_spec-1.1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/geronimo-jaspic_1.0_spec-1.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/joda-time-2.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/servlet-api-2.5.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-server-resourcemanager-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/slf4j-api-1.7.21.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hadoop-yarn-server-web-proxy-2.7.3.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/stax-api-1.0.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hbase-annotations-1.1.2.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/storm-autocreds-1.1.0.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-hcatalog-core-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/protobuf-java-2.5.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-hcatalog-server-extensions-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/snappy-java-1.0.4.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-hcatalog-streaming-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/stringtemplate-3.2.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-metastore-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/transaction-api-1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-shims-0.20S-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/velocity-1.5.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-shims-0.23-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/xercesImpl-2.9.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-shims-common-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/xml-apis-1.3.04.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-shims-scheduler-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/stax-api-1.0-2.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/hive-webhcat-java-client-1.2.1000.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/zookeeper-3.4.6.2.6.4.0-91-tests.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/json-20090211.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/json-smart-1.1.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jsp-api-2.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/jsr305-3.0.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/leveldbjni-all-1.8.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/libfb303-0.9.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/libthrift-0.9.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/log4j-1.2.17.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/mail-1.4.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/metrics-core-3.1.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/metrics-json-3.1.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/metrics-jvm-3.1.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/netty-3.9.0.Final.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/netty-all-4.0.52.Final.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/nimbus-jose-jwt-3.9.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/objenesis-2.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/okhttp-2.4.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/okio-1.4.0.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/opencsv-2.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/paranamer-2.3.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/zookeeper-3.4.6.2.6.4.0-91.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/parquet-hadoop-bundle-1.8.1.jar:/usr/hdp/current/storm-supervisor/external/storm-autocreds/xmlenc-0.52.jar:/tmp/7d39e2f050b311e892b3a0369fe7cbc4.jar:/home/metron/.storm:/usr/hdp/2.6.4.0-91/storm/bin -Dstorm.jar=/tmp/7d39e2f050b311e892b3a0369fe7cbc4.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.flux.Flux --remote /usr/hcp/1.4.2.0-23/metron/flux/indexing/batch/remote.yaml --filter /usr/hcp/1.4.2.0-23/metron/config/hdfs.properties

In /usr/hcp/1.4.2.0-23/metron/config/hdfs.properties

##### Kafka #####
kafka.zk=myzkhost1.mydomain.com:2181,myzkhost2.mydomain.com:2181,myzkhost3.mydomain.com:2181
kafka.broker=mykafkahost1.mydomain.com:6667,mykafkahost2.mydomain.com:6667,mykafkahost3.mydomain.com:6667
kafka.security.protocol=SASL_PLAINTEXT

From worker.log (kafka on mykafkahost3.mydomain.com is stopped - assuming that's why it's not in bootstrap.servers?)

2018-04-29 18:54:43.340 o.a.k.c.c.ConsumerConfig Thread-15-kafkaSpout-executor[5 5] [INFO] ConsumerConfig values:
        metric.reporters = []
        metadata.max.age.ms = 300000
        partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
        reconnect.backoff.ms = 50
        sasl.kerberos.ticket.renew.window.factor = 0.8
        max.partition.fetch.bytes = 1048576
        bootstrap.servers = [mykafkahost1.mydomain.com:6668, mykafkahost2.mydomain.com:6668]
        ssl.keystore.type = JKS
        enable.auto.commit = false
        sasl.mechanism = GSSAPI
…
        ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
        ssl.keystore.location = null
        ssl.cipher.suites = null
        security.protocol = PLAINTEXTSASL
        ssl.keymanager.algorithm = SunX509
        metrics.sample.window.ms = 30000
        auto.offset.reset = latest
2018-04-29 18:54:44.075 o.a.k.c.n.Selector Thread-15-kafkaSpout-executor[5 5] [WARN] Unexpected error from mykafkahost2.mydomain.com/192.168.1.148; closing connection
org.apache.kafka.common.errors.IllegalSaslStateException: Unexpected handshake request with client mechanism GSSAPI, enabled mechanisms are [PLAIN, GSSAPI]

I did some more digging but I'm grasping at straws. It looks like kafka.security.protocol from the properties file should be piped into the yaml files. I see the discrepancy with pcap but haven't gotten that far in testing yet...

[root@metronhost workers-artifacts]# grep -r "kafka.security.protocol"  /usr/hcp/1.4.2.0-23/*
/usr/hcp/1.4.2.0-23/metron/config/enrichment.properties:kafka.security.protocol=SASL_PLAINTEXT
/usr/hcp/1.4.2.0-23/metron/config/profiler.properties:kafka.security.protocol=SASL_PLAINTEXT
/usr/hcp/1.4.2.0-23/metron/config/elasticsearch.properties:kafka.security.protocol=SASL_PLAINTEXT
/usr/hcp/1.4.2.0-23/metron/config/pcap.properties:kafka.security.protocol=PLAINTEXT
/usr/hcp/1.4.2.0-23/metron/config/hdfs.properties:kafka.security.protocol=SASL_PLAINTEXT
/usr/hcp/1.4.2.0-23/metron/flux/enrichment/remote-unified.yaml:                  - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/enrichment/remote-unified.yaml:                  - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/enrichment/remote.yaml:                  - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/enrichment/remote.yaml:                  - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/profiler/remote.yaml:                    - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/profiler/remote.yaml:                    - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/indexing/batch/remote.yaml:                    - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/indexing/batch/remote.yaml:                    - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/indexing/random_access/remote.yaml:                    - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/indexing/random_access/remote.yaml:                    - "${kafka.security.protocol}"
/usr/hcp/1.4.2.0-23/metron/flux/pcap/remote.yaml:            - "${kafka.security.protocol}"

But...the kafka brokers and port are pulled from ZooKeeper from /brokers/ids/ (KafkaUtils.java) if it's not found in ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG. I might be on a fishing expedition but this is the only place port = 6668 beisides in the PLAINTEXT endpoint.

/brokers/ids/1003

{"jmx_port":-1,"timestamp":"1524933466675","endpoints":["SASL_PLAINTEXT://mykafkahost1.mydomain.com:6667","PLAINTEXT://mykafkahost1.mydomain.com:6668"],"host":"mykafkahost1.mydomain.com","version":3,"port":6668,"rack":"/default-rack"}

/brokers/ids/1004

{"jmx_port":-1,"timestamp":"1524933461861","endpoints":["SASL_PLAINTEXT://mykafkahost2.mydomain.com:6667","PLAINTEXT://mykafkahost2.mydomain.com:6668"],"host":"mykafkahost2.mydomain.com","version":3,"port":6668,"rack":"/default-rack"}