Member since
01-08-2017
79
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1011 | 02-27-2018 09:57 PM |
12-04-2017
04:29 PM
FYI...I didn't know the name of some of my kafka consumer groups and I figured out a way to list them all and describe each of them. Hopefully, it would help someone like myself /usr/hdp/<VERSION>/kafka/bin/kafka-consumer-groups.sh --list --zookeeper <zk_host:port> | while read group; do echo $group; /usr/hdp/<VERSION>/kafka/bin/kafka-consumer-groups.sh --zookeeper <zk_host:port> --describe -group ${group}; done
... View more
12-04-2017
03:59 PM
Thank you so much @Jasper This is pretty neat! It provides offset, logsize and Lag information. I'm fairly new to Kafka as well and it looks like i'm very behind on indexing. However, from Storm UI, it didn't look like I have that many incoming for indexing topic. I was reading and they say Lag should be close to 0, which would indicate that the system is caught up. How do I get Lag down to 0? Do I need more indexing storm workers? Offset: 82387393 logSize: 326704262 Lag: 244316869 As always, thank you for your time and response.
... View more
12-04-2017
02:40 PM
Thank you very much for your response @Jasper When I tried to run kafka-consumer-offset-checker, it said that it's deprecated and I don't have Jaas configuration in place. A quick look up it's something about single sign on kerberos account? I don't know if I need that but I'll look into it and find out. will be back for more update, thank you!
... View more
12-01-2017
09:32 PM
BTW, I realized the file global.json gets overwritten by metron/config on Ambari. There's a section called global.json template Just in case someone might find it useful.
... View more
12-01-2017
09:29 PM
Hi, I have a metron cluster running with data feeds coming in. I was wondering is there a way to verify x number of documents come through kafka for a topic, then parsing, enrichment, and then indexing to ES? The reason why I'm asking is because I don't see as many documents indexed on yesterday and today (for instance) than usually everyday. Thanks in advance for feedback.
... View more
Labels:
- Labels:
-
Apache Metron
-
Apache Storm
11-30-2017
08:36 PM
Thank you so much @rmerriman shame on me I was not aware of this global.json configuration file at all.
... View more
11-29-2017
04:21 PM
Thank you @rmerriman for your response. I tried the replay route; however tcpreplay doesn't preserve the timestamp of the original packets and I have yet to find a workaround for that. I'm going to try with the second option that you suggested. I'll look into data migration between ES/HDFS clusters. Thank you!
... View more
11-28-2017
03:54 PM
Thank you for your response @rmerriman yes, we do operational testing remotely and most of the time it doesn't have access back to our home cluster. We want to capture the network traffic remotely and bring it home.
... View more
11-27-2017
03:06 PM
Hi, I have a cluster of Metron running, by default a new index is roll out each hour per topic. After a month worth of indexes, Kibana dashboard is a bit slow when querying ElasticSearch. Is there a way to configure or combine these indexes so that the query is a bit faster? I believe Metron is designed to have long term data storage, is a month worth of data too long? Have anyone uses Metron to store data longer than that or experience in tuning for the best use case in term of time or the number of Elastic indexes? Any guidance is greatly appreciated.
... View more
Labels:
- Labels:
-
Apache Metron
11-22-2017
08:39 PM
Hi, I wanted to have a portable rack with a metron cluster setup to pickup network traffic and would like to bring that data into another metron cluster (home base). Would it be possible to import/merge the data between two metron cluster? Thank you!
... View more
Labels:
- Labels:
-
Apache Metron