Cloudera Community
jknulst
user rank
user rank
Super Collaborator

Re: Metron profiler client unbale to load native h...

by Super Collaborator Super Collaborator in Support Questions
‎10-03-2017 11:52 AM
‎10-03-2017 11:52 AM
Hi Girish, First of all this is a benign warning, it does not impact Hadoop / Metron functionality at all. You should not bother too much about it. It can be solved however if you edit HADOOP_OPTS this way: export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native" ... View more

Re: Metron Parser Fail to start with Ambari 2.5 an...

by Super Collaborator Super Collaborator in Support Questions
‎10-02-2017 12:23 PM
1 Kudo
‎10-02-2017 12:23 PM
1 Kudo
@Ashikin Abdullah HCP-1.3 / Metron 0.4.1.1.3.0.0-51 should work with HDP 2.6, but testing is still ongoing, so not for production usage. ... View more

Re: Metron Parser Fail to start with Ambari 2.5 an...

by Super Collaborator Super Collaborator in Support Questions
‎10-02-2017 12:21 PM
‎10-02-2017 12:21 PM
@Ashikin Abdullah please mark the question as answered if sufficiently answered. ... View more

Re: Kafka ConsumerGroupCommand Error

by Super Collaborator Super Collaborator in Support Questions
‎09-29-2017 01:58 PM
‎09-29-2017 01:58 PM
Reason kafka-consumer-groups.sh returned "Consumergroup`console-consumer-spout1` does not exist or is rebalancing." was that the consumergroup was not active at that moment. If it is not consuming, it is not shown with this command. In that respect kafka-consumer-offset-checker.sh does a better job as it always returns the offsets, consumer active or not ... View more

Re: Kafka ConsumerGroupCommand Error

by Super Collaborator Super Collaborator in Support Questions
‎09-20-2017 09:35 PM
1 Kudo
‎09-20-2017 09:35 PM
1 Kudo
It is solved now. Turns out that the ' --security-protocol PLAINTEXTSASL ' is not honored by the kafka-consumer-groups.sh tool. But it does work if you provide the same through the --command-config flag like: echo "security.protocol=PLAINTEXTSASL" > /tmp/consgroupcmd.prop /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop But now, to my suprise it turns out that the kafka-consumer-groups.sh / kafka.admin.ConsumerGroupCommand tool does NOT recognize any consumer group of the --new-consumer type for which offsets are stored on Kafka. Good old kafka-consumer-offset-checker.sh is still the go-to tool for this (works both on zookeeper and Kafka kept offsets): /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-offset-checker.sh --zookeeper $ZOOKEEPER_HOSTS --security-protocol PLAINTEXTSASL --topic spout_test --group console-consumer-spout1 KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/2.5.5.0-157/kafka/config/kafka_client_jaas.conf [2017-09-20 21:13:30,168] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$) Group Topic Pid Offset logSize Lag Owner console-consumer-spout1 spout_test 0 36156 68201 32045 none console-consumer-spout1 spout_test 1 0 68198 68198 none console-consumer-spout1 spout_test 2 12053 68198 56145 none console-consumer-spout1 spout_test 3 0 68200 68200 none It is weird that the deprecation warning recommends to use ConsumerGroupCommand while this does not even work on the new-consumer Kafka offsets: /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --new-consumer --describe --group console-consumer-spout1 --command-config /tmp/consgroupcmd.prop KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf Consumer group `console-consumer-spout1` does not exist or is rebalancing. [2017-09-20 21:11:04,189] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin) The group 'console-consumer-spout1' does exist as can be seen in the output of the previous command... ... View more

Re: Kafka ConsumerGroupCommand Error

by Super Collaborator Super Collaborator in Support Questions
‎09-20-2017 08:44 PM
‎09-20-2017 08:44 PM
@tsharma Thanks for your suggestion. It turns out the kafka-consumer-groups.sh eventually uses : KAFKA_KERBEROS_PARAMS=-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_jaas.conf The contents of this file is: KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="kafka" principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM"; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true renewTicket=true serviceName="kafka"; }; Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/kafka.service.keytab" storeKey=true useTicketCache=false serviceName="zookeeper" principal="kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM"; }; So it does not make a difference what principal is used to kinit. Even when I take the kafka/dem-mtrn-hds-01.field.hortonworks.com@FIELD.HORTONWORKS.COM principal I get the same result: Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(dem-mtrn-hds-01.field.hortonworks.com:6667 (id: -1 rack: null), dem-mtrn-hds-04.field.hortonworks.com:6667 (id: -4 rack: null), dem-mtrn-hds-02.field.hortonworks.com:6667 (id: -2 rack: null), dem-mtrn-hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67) at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72) at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125) at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320) at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308) at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76) at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala) ... View more

Kafka ConsumerGroupCommand Error

by Super Collaborator Super Collaborator in Support Questions
‎09-18-2017 07:35 PM
‎09-18-2017 07:35 PM
Hi, I am on HDP-2.5.5 and Kafka 0.10.0. Cluster is Kerberised. When I run the new ConsumerGroupCommand I get the error below: ubuntu@hds-01:~$ /usr/hdp/2.5.5.0-157/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $BROKER_LIST --security-protocol PLAINTEXTSASL --new-consumer --describe --group spoutconsumer [2017-09-17 19:43:54,039] WARN TGT refresh thread time adjusted from Sun Sep 17 12:49:30 UTC 2017 to Sun Sep 17 19:44:54 UTC 2017 since the former is sooner than the minimum refresh interval (60 seconds) from now. (org.apache.kafka.common.security.kerberos.KerberosLogin) [2017-09-17 19:43:54,526] WARN Bootstrap broker hds-02.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) [2017-09-17 19:43:54,533] WARN Bootstrap broker hds-04.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) [2017-09-17 19:43:54,535] WARN Bootstrap broker hds-01.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) [2017-09-17 19:43:54,638] WARN Bootstrap broker hds-03.field.hortonworks.com:6667 disconnected (org.apache.kafka.clients.NetworkClient) Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(hds-02.field.hortonworks.com:6667 (id: -2 rack: null), hds-04.field.hortonworks.com:6667 (id: -4 rack: null), hds-01.field.hortonworks.com:6667 (id: -1 rack: null), hds-03.field.hortonworks.com:6667 (id: -3 rack: null)) at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67) at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72) at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125) at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:320) at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:97) at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:308) at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:76) at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala) <br> What is wrong here? The user has valid ticket. I am quite sure the --security-protocol PLAINTEXTSASL should be OK, cause it works on both a console consumer and a producer. ... View more
Labels:

Re: Metron Parser Fail to start with Ambari 2.5 an...

by Super Collaborator Super Collaborator in Support Questions
‎09-12-2017 06:50 PM
‎09-12-2017 06:50 PM
Ashikin, Officially, all versions of Metron, including 1.2.2.0, have not been released to work with anything other then HDP-2.5.x I am quite sure it does not work with HDP-2.6 since the Storm version is higher, which explains your error message. So you have to revert to a lower version of HDP. ... View more

Re: Can I use same consumer group Id for two diffe...

by Super Collaborator Super Collaborator in Support Questions
‎09-04-2017 09:17 AM
‎09-04-2017 09:17 AM
@Shailendra Lohia Please mark the question as answered if sufficiently answered, thank you ... View more

Re: Kafka with SSL

by Super Collaborator Super Collaborator in Support Questions
‎08-31-2017 09:45 PM
‎08-31-2017 09:45 PM
On rereading the Kafka docs, it a about one and the same 'server.keystore.jks' on the brokers after all. The replacement of the initially generated unsigned broker cert by the signed version of that same cert is done by importing the signed cert with the same alias, which effectively overwrites the unsigned cert by the signed one. ... View more
Contact Me
Online
Offline
Last Visited
‎12-05-2024 10:10 AM
Community Statistics
Member Since ‎08-15-2016 02:39 PM
Last Visited ‎12-05-2024 10:10 AM
Posts 189
Kudos received 63