Support Questions
Find answers, ask questions, and share your expertise

Apache metron 0.4.2 not indexing in Elasticsearch

Apache metron 0.4.2 not indexing in Elasticsearch

Hi folks.

We have been following this tutorial to configure squid proxy on Metron. However, the data arrives in the squid topology but both, the enrichments and indexing topologies are not showing any data.

I've tailed the logs and found some warnings in storm:

91664-parser-bold.png

2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15339]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15340]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15341]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15342]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15343]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15344]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15345]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15346]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15347]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15348]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15349]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15350]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15351]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15352]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15353]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15354]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15355]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15356]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15357]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15358]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15359]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15360]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15361]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15362]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15363]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15364]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15365]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15366]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15367]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15368]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15369]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15370]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15371]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15372]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15373]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15374]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15375]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15376]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15377]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15378]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15379]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15380]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15381]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15382]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15383]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15384]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15385]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15386]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15387]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15388]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15389]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15390]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15391]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15392]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15393]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15394]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15395]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15396]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15397]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15398]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15399]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15400]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15401]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15402]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15403]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15404]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15405]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15406]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15407]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15408]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15409]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15410]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15411]. Current committed Offset [5064508]

As you can see in below screenshot, the data is arriving and it's being parsed but the enrichment and indexing topologies are not showing any events.

I tried to tail logs but I cannot find anything that lead me to do further troubleshooting.

I appreciate if you can provide some feedback.