Member since
08-28-2018
9
Posts
0
Kudos Received
0
Solutions
08-13-2019
04:21 PM
Hey @Otto Fowler, thanks for getting back. Sure, I will send the question to that group, it could just be a newbie thing, but I was not able to find a log that could point to the right direction, but surely I will send the question. Thanks.
... View more
08-12-2019
02:47 PM
Hey @Otto Fowler Can you please help here ?. I ping you directly because you pushed the sample code 🙂 (https://github.com/apache/metron/blob/master/metron-stellar/stellar-3rd-party-example/src/main/java/org/apache/metron/stellar/external/TimeFunction.java ) Thanks in advance 🙂
... View more
08-12-2019
02:45 PM
Hey @Jay Bhavsar, did you manage to address this issue? I believe I'm having the same issue.
... View more
11-05-2018
07:38 PM
I deleted @Bob Van Haute answer by mistake, adding it back here. Manually setting the source.type field seems to do the job. For your case for example:
"fieldTransformations": [
{
"input": [],
"output": [
"source.type"
],
"transformation": "STELLAR",
"config": {
"source.type": "JOIN(['squid',location,country],'-')"
}
}
]
C
... View more
11-05-2018
07:38 PM
Hi @Bob Van Haute,
That seems to work, however, the elasticsearch and hdfs indices are created wit the default configuration, because the indexing configuration wont match, leading the sensor to use the default configuration (https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.6.1/runbook/content/default_configuration.html) {
"hdfs": {
"batchSize": 10,
"enabled": true,
"index": "squid"
},
"elasticsearch": {
"batchSize": 10,
"enabled": true,
"index": "squid"
},
"solr": {
"batchSize": 1,
"enabled": false,
"index": "squid"
}
}
Storm also shows a warning because it uses the default (and potential, unoptimized settings) . Question here Is it possible that above settings matches a wildcard like *squid* ? java.lang.Exception: WARNING: Default and (likely) unoptimized writer config used for hdfs writer and sensor squid.london at org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:234) at org.apache.storm.daemon.executor$fn__10193$tuple_action_fn__10195.invoke(executor.clj:730) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:462) at org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__10193$fn__10206$fn__10259.invoke(executor.clj:849) at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
... View more
10-11-2018
06:38 PM
Hi there. Were you able to address the issue described here?
... View more
10-11-2018
05:02 PM
I'm expecting the same issue. Did you manage to solve it? Any feedback is highly appreciated.
... View more
10-11-2018
02:15 AM
Hi folks.
We have been following this tutorial to configure squid proxy on Metron. However, the data arrives in the squid topology but both, the enrichments and indexing topologies are not showing any data.
I've tailed the logs and found some warnings in storm:
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15339]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15340]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15341]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15342]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15343]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15344]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15345]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15346]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15347]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15348]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15349]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15350]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15351]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15352]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15353]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15354]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15355]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15356]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15357]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15358]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15359]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15360]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15361]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15362]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15363]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15364]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15365]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15366]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15367]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15368]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15369]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15370]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15371]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15372]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15373]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15374]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15375]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15376]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15377]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15378]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15379]. Current committed Offset [5064508]
2018-10-11 01:19:19.554 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15380]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15381]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15382]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15383]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15384]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15385]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15386]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15387]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15388]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15389]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15390]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15391]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15392]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15393]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15394]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15395]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15396]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15397]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15398]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15399]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15400]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15401]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15402]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15403]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15404]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15405]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15406]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15407]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15408]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15409]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15410]. Current committed Offset [5064508]
2018-10-11 01:19:19.555 o.a.s.k.s.i.OffsetManager Thread-14-kafkaSpout-executor[4 4] [WARN] topic-partition [squid-0] has unexpected offset [15411]. Current committed Offset [5064508]
As you can see in below screenshot, the data is arriving and it's being parsed but the enrichment and indexing topologies are not showing any events. I tried to tail logs but I cannot find anything that lead me to do further troubleshooting. I appreciate if you can provide some feedback.
... View more
Labels:
09-12-2018
05:42 PM
Hi team. I need to be able to store data from two different locations (eg US and UK) from the same sensor (squid) but be able to send the data to two different indices in elasticsearch/solar/hdfs so the data is stored separately. Since the data we are sending contains a field called location, I was expecting to create an index for Solr, Elastic and HDSF dynamic using something like squid-${location}. At runtime, depending on the value coming the location field, Metron will index into a separate indice. Note that I don't really want to have a sensor per location so we could reuse as much as possible the parsing logic. This is a quite standard way in other systems like Logstash where you can create an index like squid-{location}-{country} so the indexing is dynamic. Is this possible?
... View more
