@Geoffrey Shelton Okot Yes, i'm aware of that. I just wanted to know the service that would require restart before that change so that we can plan accordingly. ... View more

I had to clean my zookeeper data directory and recreate the __consumer_offsets topic ... View more

Only brokers 1002 and 1005 are available.. Others are old brokers which were decommissioned. Topic: __consumer_offsets Partition: 0 Leader: -1 Replicas: 33,1005,30 Isr: 30 Topic: __consumer_offsets Partition: 1 Leader: -1 Replicas: 1002,30,32 Isr: 30 Topic: __consumer_offsets Partition: 2 Leader: -1 Replicas: 1005,32,33 Isr: 32 Topic: __consumer_offsets Partition: 3 Leader: -1 Replicas: 30,33,1002 Isr: 30 Topic: __consumer_offsets Partition: 4 Leader: -1 Replicas: 32,1002,1005 Isr: 32 Topic: __consumer_offsets Partition: 5 Leader: -1 Replicas: 33,30,32 Isr: 32 Topic: __consumer_offsets Partition: 6 Leader: -1 Replicas: 1002,32,33 Isr: 32 Topic: __consumer_offsets Partition: 8 Leader: -1 Replicas: 30,1002,1005 Isr: 30 Topic: __consumer_offsets Partition: 9 Leader: -1 Replicas: 32,1005,30 Isr: 30 Topic: __consumer_offsets Partition: 10 Leader: -1 Replicas: 33,32,1002 Isr: 32 Topic: __consumer_offsets Partition: 12 Leader: -1 Replicas: 1005,1002,30 Isr: 30 Topic: __consumer_offsets Partition: 13 Leader: -1 Replicas: 30,1005,32 Isr: 30 Topic: __consumer_offsets Partition: 14 Leader: -1 Replicas: 32,30,33 Isr: 30 Topic: __consumer_offsets Partition: 16 Leader: -1 Replicas: 1002,1005,30 Isr: 30 Topic: __consumer_offsets Partition: 17 Leader: -1 Replicas: 1005,30,32 Isr: 30 Topic: __consumer_offsets Partition: 18 Leader: -1 Replicas: 30,32,33 Isr: 32 Topic: __consumer_offsets Partition: 19 Leader: -1 Replicas: 32,33,1002 Isr: 32 Topic: __consumer_offsets Partition: 20 Leader: -1 Replicas: 33,1005,30 Isr: 30 Topic: __consumer_offsets Partition: 21 Leader: -1 Replicas: 1002,30,32 Isr: 32 Topic: __consumer_offsets Partition: 22 Leader: -1 Replicas: 1005,32,33 Isr: 32 Topic: __consumer_offsets Partition: 23 Leader: -1 Replicas: 30,33,1002 Isr: 30 Topic: __consumer_offsets Partition: 24 Leader: -1 Replicas: 32,1002,1005 Isr: 32 Topic: __consumer_offsets Partition: 25 Leader: -1 Replicas: 33,30,32 Isr: 30 Topic: __consumer_offsets Partition: 26 Leader: -1 Replicas: 1002,32,33 Isr: 32 Topic: __consumer_offsets Partition: 28 Leader: -1 Replicas: 30,1002,1005 Isr: 30 Topic: __consumer_offsets Partition: 29 Leader: -1 Replicas: 32,1005,30 Isr: 30 Topic: __consumer_offsets Partition: 30 Leader: -1 Replicas: 33,32,1002 Isr: 32 Topic: __consumer_offsets Partition: 32 Leader: -1 Replicas: 1005,1002,30 Isr: 30 Topic: __consumer_offsets Partition: 33 Leader: -1 Replicas: 30,1005,32 Isr: 32 Topic: __consumer_offsets Partition: 34 Leader: -1 Replicas: 32,30,33 Isr: 30 Topic: __consumer_offsets Partition: 36 Leader: -1 Replicas: 1002,1005,30 Isr: 30 Topic: __consumer_offsets Partition: 37 Leader: -1 Replicas: 1005,30,32 Isr: 32 Topic: __consumer_offsets Partition: 38 Leader: -1 Replicas: 30,32,33 Isr: 32 Topic: __consumer_offsets Partition: 39 Leader: -1 Replicas: 32,33,1002 Isr: 32 Topic: __consumer_offsets Partition: 40 Leader: -1 Replicas: 33,1005,30 Isr: 30 Topic: __consumer_offsets Partition: 41 Leader: -1 Replicas: 1002,30,32 Isr: 30 Topic: __consumer_offsets Partition: 42 Leader: -1 Replicas: 1005,32,33 Isr: 32 Topic: __consumer_offsets Partition: 43 Leader: -1 Replicas: 30,33,1002 Isr: 30 Topic: __consumer_offsets Partition: 44 Leader: -1 Replicas: 32,1002,1005 Isr: 32 Topic: __consumer_offsets Partition: 45 Leader: -1 Replicas: 33,30,32 Isr: 30 Topic: __consumer_offsets Partition: 46 Leader: -1 Replicas: 1002,32,33 Isr: 32 Topic: __consumer_offsets Partition: 48 Leader: -1 Replicas: 30,1002,1005 Isr: 30 Topic: __consumer_offsets Partition: 49 Leader: -1 Replicas: 32,1005,30 Isr: 30 ... View more

Hortonworks support said it supports 9.4. ... View more

@Geoffrey Shelton Okot Ofcourse, we did that and its working and there is no warning in the ambari-server-check-database.log about the database version but since its not mentioned in support matrix, we are on crossroads. ... View more

@Geoffrey Shelton Okot By any chance does running ambari on 9.5 have any issue? We have a test cluster with ambari running on 9.6.8 with no issues till now. Want to make sure it doesn't have issues if used i Production. ... View more

Hi @Geoffrey Shelton Okot Yes, I have seen that, We want to install HDF ans SAM, Schema registry require 9.5. So was wondering if ambari supports it or not rather than having two versions of PostGres. ... View more

@SIOKEK YIM Do you mean all my topic creation, producing and consuming has to be done with kafka user? I'm not sure if kafka has any impersonation..!! Tried doing using kafka keytab. Below is the topic created with kafka service keytab. But its the same issue Topic:Hello1 PartitionCount:2 ReplicationFactor:2 Configs:min.insync.replicas=2 Topic: Hello1 Partition: 0 Leader: 35 Replicas: 35,38 Isr: 35,38 Topic: Hello1 Partition: 1 Leader: 38 Replicas: 38,33 Isr: 38,33 Producer: =======kafka-verifiable-producer.sh --topic Hello1 --broker-list server1.kafka2.pre.corp:9092,server2.kafka2.pre.corp:9092 --producer.config client.properties 17/11/23 09:58:07 INFO utils.AppInfoParser: Kafka version : 0.10.2-kafka-2.2.0 17/11/23 09:58:07 INFO utils.AppInfoParser: Kafka commitId : unknown 1 2 3 4 45 17/11/23 09:58:25 INFO producer.KafkaProducer: Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. {"name":"shutdown_complete"} {"sent":1,"name":"tool_data","avg_throughput":0.0,"target_throughput":-1,"acked":0} Consumer: kafka-console-consumer --topic Hello1 --from-beginning --zookeeper server.kafka2.pre.corp:2181,server.kafka2.pre.corp:2181/kafka2dc1pre --consumer.config consumer.properties 17/11/23 09:59:04 INFO utils.ZKCheckedEphemeral: Creating /consumers/console-consumer-59653/ids/console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7 (is it secure? false) 17/11/23 09:59:04 INFO utils.ZKCheckedEphemeral: Result of znode creation is: OK . . 17/11/23 09:59:05 INFO consumer.ZookeeperConsumerConnector: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7], end rebalancing consumer console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7 try #0 17/11/23 09:59:05 INFO consumer.ZookeeperConsumerConnector: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7], Creating topic event watcher for topics Hello1 17/11/23 09:59:05 INFO consumer.ZookeeperConsumerConnector: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7], Topics to consume = ArrayBuffer(Hello1) 17/11/23 09:59:05 WARN consumer.ConsumerFetcherManager$LeaderFinderThread: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7-leader-finder-thread], Failed to find leader for Set(Hello1-0, Hello1-1) kafka.common.BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker 33 at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1$anonfun$apply$5.apply(ClientUtils.scala:146) at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1$anonfun$apply$5.apply(ClientUtils.scala:146) at scala.Option.getOrElse(Option.scala:121) at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:146) at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1.apply(ClientUtils.scala:142) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.AbstractTraversable.map(Traversable.scala:104) at kafka.client.ClientUtils$.getPlaintextBrokerEndPoints(ClientUtils.scala:142) at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) 17/11/23 09:59:05 INFO consumer.ConsumerFetcherManager: [ConsumerFetcherManager-1511431144680] Added fetcher for partitions ArrayBuffer() 17/11/23 09:59:05 WARN consumer.ConsumerFetcherManager$LeaderFinderThread: [console-consumer-59653_server1.kafka2.pre.corp-1511431144657-52e4b1e7-leader-finder-thread], Failed to find leader for Set(Hello1-0, Hello1-1) kafka.common.BrokerEndPointNotAvailableException: End point with security protocol PLAINTEXT not found for broker 33 at kafka.client.ClientUtils$anonfun$getPlaintextBrokerEndPoints$1$anonfun$apply$5.apply(ClientUtils.scala:146) ... View more

The issue actually seems to be with the kafka root in zookeeper. Initially the kafka root was / and then we changed it to /kafkapre in zookeeper. But kafka seems to be searching for / itself still. 2017-11-23 09:16:12,539 WARN nl.techop.kafka.dao.zookeeper.KafkaZkClient: KeeperErrorCode = NoNode for /brokers/topics/TEST/partitions/1/state 2017-11-23 09:17:42,548 WARN nl.techop.kafka.dao.zookeeper.KafkaZkClient: KeeperErrorCode = NoNode for /brokers/topics/TEST/partitions/1/state 2017-11-23 09:19:12,533 WARN nl.techop.kafka.dao.zookeeper.KafkaZkClient: KeeperErrorCode = NoNode for /brokers/topics/TEST/partitions/1/state ... View more

@SIOKEK YIM Do i need to be using kafka keytab?? ... View more

@Aditya Sirna Thanks for the prompt reply. Yes, we did all the mentioned above. ... View more

@aditya im using a standalone Kafka. ... View more

Below are the errors i see in producer. 17/11/22 12:43:01 ERROR internals.ErrorLoggingCallback: Error when sending message to topic TEST with key: null, value: 4 bytes with error: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. 17/11/22 12:44:01 ERROR internals.ErrorLoggingCallback: Error when sending message to topic TEST with key: null, value: 2 bytes with error: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. 17/11/22 12:45:01 ERROR internals.ErrorLoggingCallback: Error when sending message to topic TEST with key: null, value: 5 bytes with error: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. 17/11/22 12:46:01 ERROR internals.ErrorLoggingCallback: Error when sending message to topic TEST with key: null, value: 4 bytes with error: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. ... View more

hi @Geoffrey Shelton Okot We are using standard MIT setup. Its just that we have 3 KDC servers nothing else defers in the configuration process. In fact i created the trust setup from the beginning and its working from Cluster A. Since Cluster B doesn't have hdfs, we cant access Cluster A hdfs with below error. cabig02@srvbdadvlsk32:~$ hdfs dfs -ls hdfs://srvbdadvlsk21.devoperational1.xxxxxxx.pre.corp:8020/ ls: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] Warning: fs.defaultFS is not set when running "ls" command. cabig02@srvbdadvlsk32:~$ ... View more