@Nasheb Ismaily, Double check your configuration. I know you already know this, but for the sake of a complete answer, here's how to configure FIFO. The capacity scheduler queues can be configured for fifo or fair based via Ambari's Yarn Queue Manager (top right button). The default is fifo. Via Ambari - Yarn Capacity Scheduler Queue configuration: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_performance_tuning/content/section_create_configure_yarn_capacity_scheduler_queues.html Manually: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_yarn_resource_mgt/content/flexible_scheduling_policies.html Also, the Yarn Fair Scheduler can be configured for FIFO: https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/FairScheduler.html “schedulingPolicy: to set the scheduling policy of any queue. The allowed values are “fifo”/“fair”/“drf” or any class that extends” ... View more

@Nasheb Ismaily, according to what I read, applications are in fifo order, according to the time of submission. If you submit them back to back very quickly, is it possible the timestamps are identical and arrived at the "same time"? ... View more

I want to correct what I said above. On the client, kafka.consumer exists, but I was looking on the broker. And, the answer below is correct (newly named metric/Mbean is on the new consumer). ... View more

@schintalapani - Thanks! That's what I needed to know! ... View more

@schintalapani I believe it is the new am using the interface provided by this (http://kafka.apache.org/documentation.html#newconsumerapi). <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.9.0.0</version> </dependency> org.apache.kafka.clients.consumer.KafkaConsumer<String,String> However, I may be missing a fundamental point. I have mostly been looking at mbeans from the broker, not the client. I did run the client but still did not see kafka.consumer mbeans there either. ... View more

Thanks for the info @swagle. We were able to convince the customer to upgrade to Ambari 2.2.2. They were thrilled with the fact that you can add widgets directly in the UI, and the fact that Grafana is available is icing on the cake. Incidentally, here is the JIRA for adding this for Kafka and Storm: https://issues.apache.org/jira/browse/AMBARI-14941 ... View more

These files have stuff like this in them, so how do I get this info in Ambari? { "KAFKA_BROKER": { "Component": [ { "type": "ganglia", "metrics": { "default": { "metrics/jvm/uptime": { "metric": "jvm.uptime", "pointInTime": true, "temporal": true }, /* -- SNIP -- */ "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesInPerSec/count": { "metric": "kafka.server.BrokerTopicMetrics.BytesInPerSec.count", "pointInTime": true, "temporal": true }, "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/1MinuteRate": { "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.1MinuteRate", "pointInTime": true, "temporal": true }, "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/5MinuteRate": { "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.5MinuteRate", "pointInTime": true, "temporal": true }, "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/15MinuteRate": { "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.15MinuteRate", "pointInTime": true, "temporal": true }, "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/meanRate": { "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.meanRate", "pointInTime": true, "temporal": true }, "metrics/kafka/server/BrokerTopicMetrics/AllTopicsBytesOutPerSec/count": { "metric": "kafka.server.BrokerTopicMetrics.BytesOutPerSec.count", "pointInTime": true, "temporal": true }, "metrics/kafka/controller/KafkaController/ActiveControllerCount": { "metric": "kafka.controller.KafkaController.ActiveControllerCount", "pointInTime": true, "temporal": true }, ... View more