About nnat25191

nnat25191 · ‎12-04-2017

FYI...I didn't know the name of some of my kafka consumer groups and I figured out a way to list them all and describe each of them. Hopefully, it would help someone like myself /usr/hdp/<VERSION>/kafka/bin/kafka-consumer-groups.sh --list --zookeeper <zk_host:port> | while read group; do echo $group; /usr/hdp/<VERSION>/kafka/bin/kafka-consumer-groups.sh --zookeeper <zk_host:port> --describe -group ${group}; done

nnat25191 · ‎12-04-2017

Thank you so much @Jasper This is pretty neat! It provides offset, logsize and Lag information. I'm fairly new to Kafka as well and it looks like i'm very behind on indexing. However, from Storm UI, it didn't look like I have that many incoming for indexing topic. I was reading and they say Lag should be close to 0, which would indicate that the system is caught up. How do I get Lag down to 0? Do I need more indexing storm workers? Offset: 82387393 logSize: 326704262 Lag: 244316869 As always, thank you for your time and response.

nnat25191 · ‎12-04-2017

Thank you very much for your response @Jasper When I tried to run kafka-consumer-offset-checker, it said that it's deprecated and I don't have Jaas configuration in place. A quick look up it's something about single sign on kerberos account? I don't know if I need that but I'll look into it and find out. will be back for more update, thank you!

nnat25191 · ‎12-01-2017

BTW, I realized the file global.json gets overwritten by metron/config on Ambari. There's a section called global.json template Just in case someone might find it useful.

nnat25191 · ‎12-01-2017

Hi, I have a metron cluster running with data feeds coming in. I was wondering is there a way to verify x number of documents come through kafka for a topic, then parsing, enrichment, and then indexing to ES? The reason why I'm asking is because I don't see as many documents indexed on yesterday and today (for instance) than usually everyday. Thanks in advance for feedback.

nnat25191 · ‎11-30-2017

Thank you so much @rmerriman shame on me I was not aware of this global.json configuration file at all.

nnat25191 · ‎11-29-2017

Thank you @rmerriman for your response. I tried the replay route; however tcpreplay doesn't preserve the timestamp of the original packets and I have yet to find a workaround for that. I'm going to try with the second option that you suggested. I'll look into data migration between ES/HDFS clusters. Thank you!

nnat25191 · ‎11-28-2017

Thank you for your response @rmerriman yes, we do operational testing remotely and most of the time it doesn't have access back to our home cluster. We want to capture the network traffic remotely and bring it home.

nnat25191 · ‎11-27-2017

Hi, I have a cluster of Metron running, by default a new index is roll out each hour per topic. After a month worth of indexes, Kibana dashboard is a bit slow when querying ElasticSearch. Is there a way to configure or combine these indexes so that the query is a bit faster? I believe Metron is designed to have long term data storage, is a month worth of data too long? Have anyone uses Metron to store data longer than that or experience in tuning for the best use case in term of time or the number of Elastic indexes? Any guidance is greatly appreciated.

nnat25191 · ‎11-22-2017

Hi, I wanted to have a portable rack with a metron cluster setup to pickup network traffic and would like to bring that data into another metron cluster (home base). Would it be possible to import/merge the data between two metron cluster? Thank you!

Online	Offline
Last Visited	‎06-13-2019 03:53 PM

Member Since	‎01-08-2017 10:33 PM
Last Visited	‎06-13-2019 03:53 PM
Posts	79
Kudos received	6

Cloudera Community

Re: Metron 4.2.0 deployment to AWS

Re: How do I verify all of my data from enrichment...

Re: How do I verify all of my data from enrichment...

Re: How do I verify all of my data from enrichment...

Re: Metron ElasticSearch too many indexes

How do I verify all of my data from enrichment mad...

Re: Metron ElasticSearch too many indexes

Re: import data from one metron cluster to another

Re: import data from one metron cluster to another

Metron ElasticSearch too many indexes

import data from one metron cluster to another