Dear, After upgraded from HDP 2.6.4 to 2.6.5 in my LAB to check all components before upgrade DEV and PROD cluster, I realized that Atlas was not working properly missing the informations for lineage. Going deeply I found an issue on Kafka that kafka console consumer is not getting the messages through these simple commands below: hduser@> kafka-console-consumer.sh --bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic kennon_2 --security-protocol SASL_PLAINTEXT --from-beginning hduser@> kafka-console-consumer.sh --bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic ATLAS_HOOK --security-protocol SASL_PLAINTEXT --from-beginning hduser@> kafka-console-consumer.sh --bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic ATLAS_ENTITIES --security-protocol SASL_PLAINTEXT --from-beginning There is no message consuming, but I checked the producer and all the messages are going to the topic. If I put the option '--partition 0' for example I just see the messages from this partition and If I change --bootstrap-server to --zookeeper I can see almost all messages, yes I said almost not all, like 80% of the messages. In this case there is no log error because there is a connection established and the application just waiting to consume the message. hduser@> kafka-console-consumer.sh ---bootstrap-server 10.0.20.101:6667,10.0.20.102:6667,10.0.20.103:6667 --topic kennon_2 --security-protocol SASL_PLAINTEXT --from-beginning --partition 0 or hduser@> kafka-console-consumer.sh --zookeeper 10.0.20.101:2181,10.0.20.102:2181,10.0.20.103:2181/kafka --topic kennon_2 --security-protocol SASL_PLAINTEXT --from-beginning I have another LAB that I deployed a little cluster from the scratch to test HDP 2.6.5 and I'm facing the same issue with Kafka. The concern is If we upgrade the version will take a little time to realized that all applications consuming kafka topics are not working properly, including Apache Atlas that is not doing the lineage or consuming the ATLAS TOPICS. In the last 2 previous (HDP 2.6.3 and 2.6.4) versions are working fine, without any problem. Taking a look to the release notes (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/new_features.html) KAFKA is a new version and there are some patch fixes. Kafka This release provides Kafka 1.0.0 and the following Apache patches. KAFKA-4827: Kafka connect: error with special characters in connector name. KAFKA-6118: Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials. KAFKA-6156: JmxReporter can't handle windows style directory paths. KAFKA-6164: ClientQuotaManager threads prevent shutdown when encountering an error loading logs. KAFKA-6167: Timestamp on streams directory contains a colon, which is an illegal character. KAFKA-6179: RecordQueue.clear() does not clear MinTimestampTracker's maintained list. KAFKA-6185: Selector memory leak with high likelihood of OOM in case of down conversion. KAFKA-6190: GlobalKTable never finishes restoring when consuming transactional messages. KAFKA-6210: IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version. KAFKA-6214: Using standby replicas with an in memory state store causes Streams to crash. KAFKA-6215: KafkaStreamsTest fails in trunk. KAFKA-6238: Issues with protocol version when applying a rolling upgrade to 1.0.0. KAFKA-6260: AbstractCoordinator not clearly handles NULL Exception. KAFKA-6261: Request logging throws exception if acks=0. KAFKA-6274: Improve KTable Source state store auto-generated names. HDP 2.6.4 provided Kafka 0.10.1 with no additional Apache patches. HDP 2.6.3 provided Kafka 0.10.1 and the following Apache patches: KAFKA-4360: Controller may deadLock when autoLeaderRebalance encounter zk expired. HDP 2.6.2 provided Kafka 0.10.1 with no additional Apache patches. HDP 2.6.1 provided Kafka 0.10.1 with no additional Apache patches. HDP 2.6.0 provided Kafka 0.10.1 with no additional Apache patches. Please guys If do you have some thoughts I'll appreciate. Kennon ... View more

Hello, I'm having problem with phoenix flume sink to upsert data into phoenix table only with records separated by binary caracter (\001). I tried a lot of regex for binary 001 separator but I don't having any success. When I change the records with "|" (pipe) that works fine and all data goes to the right field/column. I need use the binary character as separator, because some values have pipe or another unusual characters in the columns from the source. I'm using "phoenix-flume-4.7.0-HBase-1.1.jar" in the HDP 2.5.3. See below the Flume Phoenix config, regex that I tried and example of the data. -----------------------------------BEGIN FLUME CONF------------ #(APACHE PHOENIX - LOG CORP - TEST - flume conf) #(APACHE PHOENIX - LOG CORP - TEST) logcptv_test.sources = logcptv-source-test logcptv_test.channels = memoryChannel logcptv_test.sinks = phoenix-sink #configuration for channel logcptv_test.channels.memoryChannel.type=memory logcptv_test.channels.memoryChannel.transactionCapacity=1000 logcptv_test.channels.memoryChannel.capacity = 10000 #configTEMP Source logcptv_test.sources.logcptv-source-test.type=spooldir logcptv_test.sources.logcptv-source-test.spoolDir=/flumeSpool #configuration for source #logcptv_test.sources.logcptv-source-test.type = avro logcptv_test.sources.logcptv-source-test.bind = 0.0.0.0 logcptv_test.sources.logcptv-source-test.port = 4141 logcptv_test.sources.logcptv-source-test.channels = memoryChannel #configuration for sink #logcptv_test.sinks.phoenix-sink.type=com.salesforce.phoenix.flume.sink.PhoenixSink logcptv_test.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink logcptv_test.sinks.phoenix-sink.channel = memoryChannel logcptv_test.sinks.phoenix-sink.batchSize = 100 logcptv_test.sinks.phoenix-sink.table = LOGCPTV_TEST_6 logcptv_test.sinks.phoenix-sink.zookeeperQuorum=lxl1big32 #logcptv_test.sinks.phoenix-sink.serializer.rowkeyType=uuid logcptv_test.sinks.phoenix-sink.ddl=CREATE TABLE IF NOT EXISTS LOGCPTV_TEST_6(NR_VRS_HDR TINYINT,TX_CHV_PPL_DRV_CRIO VARCHAR(1),TX_CHV_PPL_IDFR_UNCO VARCHAR(19),TX_CHV_PPL_ETP TINYINT,TX_CHV_PPL_SEQL_ETP TINYINT,TS TIMESTAMP,AMB_GERADOR_EVT VARCHAR(1) CONSTRAINT pk PRIMARY KEY (TS)) logcptv_test.sinks.phoenix-sink.serializer = REGEX logcptv_test.sinks.phoenix-sink.serializer.regex=([^\\001]*)\\001([^\\001]*)\\001([^\\001]*)\\001([^\\001]*)\\001([^\\001]*)\\001([^\\001]*)\\001([^\\001]*) #(For records separated by "|" pipe caracter - that works fine) #logcptv_test.sinks.phoenix-sink.serializer.regex=([^\\|]*)\\|([^\\|]*)\\|([^\\|]*)\\|([^\\|]*)\\|([^\\|]*)\\|([^\\|]*)\\|([^\\|]*)\\|([^\\|]*) logcptv_test.sinks.phoenix-sink.serializer.columns= NR_VRS_HDR,TX_CHV_PPL_DRV_CRIO,TX_CHV_PPL_IDFR_UNCO,TX_CHV_PPL_ETP,TX_CHV_PPL_SEQL_ETP,TS,AMB_GERADOR_EVT #other regex tested for binary separator #logcptv_agent1.sinks.phoenix-sink.serializer.regex=([^
]*)
([^
]*)
([^
]*)
#logcptv_agent1.sinks.phoenix-sink.serializer.regex=([^\u0001]*)\u0001([^\u0001]*)\u0001([^\u0001]*)\u0001 ------------------------END FLUME CONF--------------- # Example of two record using binary separator 01
W
%rfTYt;8dzRTÂT5gTp-
1
1
2017-05-17 14:58:34.675000
W 01
W
%rVght;7dzPTÂTAk5p-
1
1
2017-05-17 14:58:35.335000
W Best Regards, Kennon ... View more