Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache metron 0.4.2 not indexing in Elasticsearch

Highlighted

Apache metron 0.4.2 not indexing in Elasticsearch

Hi folks.

We have been following this tutorial to configure squid proxy on Metron. However, the data arrives in the squid topology but both, the enrichments and indexing topologies are not showing any data.

I've tailed the logs and found some warnings in storm:

91664-parser-bold.png

2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15339]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15340]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15341]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15342]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15343]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15344]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15345]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15346]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15347]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15348]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15349]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15350]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15351]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15352]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15353]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15354]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15355]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15356]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15357]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15358]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15359]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15360]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15361]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15362]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15363]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15364]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15365]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15366]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15367]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15368]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15369]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15370]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15371]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15372]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15373]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15374]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15375]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15376]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15377]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15378]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15379]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15380]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15381]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15382]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15383]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15384]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15385]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15386]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15387]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15388]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15389]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15390]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15391]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15392]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15393]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15394]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15395]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15396]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15397]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15398]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15399]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15400]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15401]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15402]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15403]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15404]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15405]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15406]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15407]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15408]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15409]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15410]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15411]. Current committed Offset [5064508]

As you can see in below screenshot, the data is arriving and it's being parsed but the enrichment and indexing topologies are not showing any events.

I tried to tail logs but I cannot find anything that lead me to do further troubleshooting.

I appreciate if you can provide some feedback.

Don't have an account?
Coming from Hortonworks? Activate your account here