Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Metron not indexing in elasticsearch

Highlighted

Metron not indexing in elasticsearch

New Contributor

Hi all i am trying to acheive checkpoint logs into metron. I am using the following flow checkpoint --> nifi --> kafka -> logstash -> kafka (new topic) -> Logstash metron parser

Here are my files

Logstash conf

input {
 kafka{
   bootstrap_servers => "10.122.62.98:6667,10.122.62.99:6667"
   topics => "checkpoint"
   client_id => "checkpointwireless"
   tags => "checkpointfw"
}
}




filter {


if "checkpointfw" in [tags] {
syslog_pri { }


 if "checkpointfw" in [tags]  {
kv {
source => "message"
}


}
}
}




output {
if "checkpointfw" in [tags]   {
#stdout { codec => json_lines}
kafka {
 codec => rubydebug
 bootstrap_servers => "10.122.62.98:6667,10.122.62.99:6667"
 topic_id => "checkpoint_plain"
}
}
}


2 REPLIES 2

Re: Metron not indexing in elasticsearch

New Contributor

Here are my json files for indexing and parsing

{"parserClassName":"org.apache.metron.parsers.logstash.BasicLogstashParser","filterClassName":null,"sensorTopic":"checkpoint_wifi","writerClassName":null,"errorWriterClassName":null,"invalidWriterClassName":null,"readMetadata":false,"mergeMetadata":false,"numWorkers":null,"numAckers":null,"spoutParallelism":1,"spoutNumTasks":1,"parserParallelism":1,"parserNumTasks":1,"errorWriterParallelism":1,"errorWriterNumTasks":1,"spoutConfig":{},"securityProtocol":null,"stormConfig":{},"parserConfig":{},"fieldTransformations":[]}

Re: Metron not indexing in elasticsearch

New Contributor
{"hdfs":{"batchSize":1,"enabled":true,"index":"checkpoint"},"elasticsearch":{"batchSize":1,"enabled":true,"index":"checkpoint"},"solr":{"batchSize":1,"enabled":false,"index":"checkpoint_wifi"}}
Don't have an account?
Coming from Hortonworks? Activate your account here