Dear, After upgraded from HDP 2.6.4 to 2.6.5 in my LAB to check all components before upgrade DEV and PROD cluster, I realized that Atlas was not working properly missing the informations for lineage. Going deeply I found an issue on Kafka that kafka console consumer is not getting the messages through these simple commands below: hduser@> kafka-console-consumer.sh --bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic kennon_2 --security-protocol SASL_PLAINTEXT --from-beginning hduser@> kafka-console-consumer.sh --bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic ATLAS_HOOK --security-protocol SASL_PLAINTEXT --from-beginning hduser@> kafka-console-consumer.sh --bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic ATLAS_ENTITIES --security-protocol SASL_PLAINTEXT --from-beginning There is no message consuming, but I checked the producer and all the messages are going to the topic. If I put the option '--partition 0' for example I just see the messages from this partition and If I change --bootstrap-server to --zookeeper I can see almost all messages, yes I said almost not all, like 80% of the messages. In this case there is no log error because there is a connection established and the application just waiting to consume the message. hduser@> kafka-console-consumer.sh ---bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic kennon_2 --security-protocol SASL_PLAINTEXT --from-beginning --partition 0 or hduser@> kafka-console-consumer.sh --zookeeper 10.0.20.101:2181,10.0.20.102:2181,10.0.20.103:2181/kafka --topic kennon_2 --security-protocol SASL_PLAINTEXT --from-beginning I have another LAB that I deployed a little cluster from the scratch to test HDP 2.6.5 and I'm facing the same issue with Kafka. The concern is If we upgrade the version will take a little time to realized that all applications consuming kafka topics are not working properly, including Apache Atlas that is not doing the lineage or consuming the ATLAS TOPICS. In the last 2 previous (HDP 2.6.3 and 2.6.4) versions are working fine, without any problem. Taking a look to the release notes (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/new_features.html) KAFKA is a new version and there are some patch fixes. Kafka This release provides Kafka 1.0.0 and the following Apache patches. KAFKA-4827: Kafka connect: error with special characters in connector name. KAFKA-6118: Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials. KAFKA-6156: JmxReporter can't handle windows style directory paths. KAFKA-6164: ClientQuotaManager threads prevent shutdown when encountering an error loading logs. KAFKA-6167: Timestamp on streams directory contains a colon, which is an illegal character. KAFKA-6179: RecordQueue.clear() does not clear MinTimestampTracker's maintained list. KAFKA-6185: Selector memory leak with high likelihood of OOM in case of down conversion. KAFKA-6190: GlobalKTable never finishes restoring when consuming transactional messages. KAFKA-6210: IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version. KAFKA-6214: Using standby replicas with an in memory state store causes Streams to crash. KAFKA-6215: KafkaStreamsTest fails in trunk. KAFKA-6238: Issues with protocol version when applying a rolling upgrade to 1.0.0. KAFKA-6260: AbstractCoordinator not clearly handles NULL Exception. KAFKA-6261: Request logging throws exception if acks=0. KAFKA-6274: Improve KTable Source state store auto-generated names. HDP 2.6.4 provided Kafka 0.10.1 with no additional Apache patches. HDP 2.6.3 provided Kafka 0.10.1 and the following Apache patches: KAFKA-4360: Controller may deadLock when autoLeaderRebalance encounter zk expired. HDP 2.6.2 provided Kafka 0.10.1 with no additional Apache patches. HDP 2.6.1 provided Kafka 0.10.1 with no additional Apache patches. HDP 2.6.0 provided Kafka 0.10.1 with no additional Apache patches. Please guys If do you have some thoughts I'll appreciate. Kennon ... View more