Created 12-13-2017 10:00 AM
I have ingested squid telemtery into metron and can see alerts in the kibana dashboard (I have injested threat intel and setup risk scoring). However the metron alerts UI is empty. Do I need additional configuration for these alerts to show up in the alerts UI?
Created 12-13-2017 11:09 PM
Only 'alerts' will appear in the Alerts UI. So what is an alert then, you ask?
Well, not all telemetry in Metron is treated as an alert. Only telemetry that is specifically marked with a field { "is_alert": "true" } is treated as an alert. This gives the user the flexibility to define which telemetry will go through additional threat triage processing.
In your case, the Squid telemetry does not have this field and so is not treated as an alert. For testing purposes, you can add this field to your Squid telemetry by creating a simple enrichment that adds the field "is_alert" and sets it to "true".
Hope this makes sense.
Created 12-14-2017 12:28 AM
The squid telemetry does have is_alert true.that is the first thing I checked. I have done threat intel enrichment and is_alert is set to true based on the threat intel
Created 12-14-2017 12:49 AM
Do you see anything in the Alerts UI? Or is just this specific Squid data that is missing? What version of Metron are you running? How did you deploy Metron?
Created 12-14-2017 02:53 AM
Nothing at all in the alerts UI. I see 5 alerts in kibanan and 12 non alert events. But alerts UI is empty. I am running 1.3.1.0-37 installed through the ambari mpack
Created 12-14-2017 02:54 AM
hcp 1.3.1.0-37
Created 12-15-2017 02:55 PM
I've been facing same issue, metron-alarms-ui wasn't receiving any alarms, after applying changes to template and adding field "alert", type nested, metron-alarms-ui started receiving some messages(little bit messed up, which is next thing to understand), but looking at your template, i notice that field is_alert is type string, while in my schema it is boolean, maybe this makes some problems.
Created 12-14-2017 02:50 AM
I am running 1.3.1.0-37. Deployed it through the ambari mpack
Created 12-14-2017 04:46 AM
You might be running into METRON-1283, which is fixed in more recent versions of HCP. You can confirm as follows - Go to http://<es-master-node>:9200/squid*/_mappings, and look for an "alerts" field. If you are missing this field from the mapping, then it is likely the reason for Alerts UI to be empty.
In order to fix the issue, follow these steps:
* Clear all existing Elasticsearch indices
* Go to Ambari UI -> Services -> Metron -> 'Service Actions' dropdown -> Elasticsearch Template Install
* Re-ingest data into 'squid' telemetry.
And you should now be able to see entries in the Alerts UI.
Created 12-14-2017 09:36 AM
I tried re-installing the elastic template. Still no luck
Created 12-14-2017 09:51 AM
Can you paste the output when you open 'http://<es-master-node>:9200/squid*/_mappings' ?
Created 12-14-2017 10:18 AM
properties | |
action | |
type | "string" |
index | "not_analyzed" |
adapter:geoadapter:begin:ts | |
type | "string" |
adapter:geoadapter:end:ts | |
type | "string" |
adapter:stellaradapter:begin:ts | |
type | "string" |
adapter:stellaradapter:end:ts | |
type | "string" |
bytes | |
type | "integer" |
code | |
type | "string" |
index | "not_analyzed" |
domain_without_subdomains | |
type | "string" |
index | "not_analyzed" |
elapsed | |
type | "integer" |
enrichmentjoinbolt:joiner:ts | |
type | "string" |
enrichments:geo:ip_dst_addr:city | |
type | "string" |
enrichments:geo:ip_dst_addr:country | |
type | "string" |
enrichments:geo:ip_dst_addr:dmaCode | |
type | "string" |
enrichments:geo:ip_dst_addr:latitude | |
type | "string" |
enrichments:geo:ip_dst_addr:locID | |
type | "string" |
enrichments:geo:ip_dst_addr:location_point | |
type | "string" |
enrichments:geo:ip_dst_addr:longitude | |
type | "string" |
enrichments:geo:ip_dst_addr:postalCode | |
type | "string" |
enrichmentsplitterbolt:splitter:begin:ts | |
type | "string" |
enrichmentsplitterbolt:splitter:end:ts | |
type | "string" |
extra | |
type | "string" |
full_hostname | |
type | "string" |
index | "not_analyzed" |
guid | |
type | "string" |
ip_dst_addr | |
type | "ip" |
ip_dst_port | |
type | "integer" |
ip_src_addr | |
type | "ip" |
ip_src_port | |
type | "integer" |
is_malicious | |
type | "string" |
method | |
type | "string" |
index | "not_analyzed" |
original_string | |
type | "string" |
source:type | |
type | "string" |
index | "not_analyzed" |
threatinteljoinbolt:joiner:ts | |
type | "string" |
threatintelsplitterbolt:splitter:begin:ts | |
type | "string" |
threatintelsplitterbolt:splitter:end:ts | |
type | "string" |
timestamp | |
type | "date" |
format | "epoch_millis" |
url | |
type | "string" |
Created 12-14-2017 10:22 AM
{"squid_index_2017.12.14.09":{"mappings":{"squid_doc":{"_timestamp":{"enabled":true},"properties":{"action":{"type":"string","index":"not_analyzed"},"adapter:simplehbaseadapter:begin:ts":{"type":"string"},"adapter:simplehbaseadapter:end:ts":{"type":"string"},"adapter:threatinteladapter:begin:ts":{"type":"string"},"adapter:threatinteladapter:end:ts":{"type":"string"},"alert":{"type":"nested"},"bytes":{"type":"integer"},"code":{"type":"string","index":"not_analyzed"},"domain_without_subdomains":{"type":"string","index":"not_analyzed"},"elapsed":{"type":"integer"},"enrichmentjoinbolt:joiner:ts":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:domain":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:domain_created_timestamp":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:home_country":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:owner":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:registrar":{"type":"string"},"enrichmentsplitterbolt:splitter:begin:ts":{"type":"string"},"enrichmentsplitterbolt:splitter:end:ts":{"type":"string"},"full_hostname":{"type":"string","index":"not_analyzed"},"guid":{"type":"string"},"ip_dst_addr":{"type":"string","index":"not_analyzed"},"ip_src_addr":{"type":"string"},"is_alert":{"type":"string"},"method":{"type":"string","index":"not_analyzed"},"original_string":{"type":"string"},"source:type":{"type":"string","index":"not_analyzed"},"threat:triage:rules:0:comment":{"type":"string"},"threat:triage:rules:0:name":{"type":"string"},"threat:triage:rules:0:reason":{"type":"string"},"threat:triage:rules:0:score":{"type":"long"},"threat:triage:rules:1:comment":{"type":"string"},"threat:triage:rules:1:name":{"type":"string"},"threat:triage:rules:1:reason":{"type":"string"},"threat:triage:rules:1:score":{"type":"long"},"threat:triage:score":{"type":"double"},"threatinteljoinbolt:joiner:ts":{"type":"string"},"threatintels:hbaseThreatIntel:domain_without_subdomains:zeusList":{"type":"string"},"threatintelsplitterbolt:splitter:begin:ts":{"type":"string"},"threatintelsplitterbolt:splitter:end:ts":{"type":"string"},"timestamp":{"type":"date","format":"epoch_millis"},"url":{"type":"string"}}}}}}
Created 12-14-2017 10:33 AM
Hmm... this seems about right to me. It should be working, in principle.
In my environment, I have seen snort alerts working readily. Can you try ingesting the following sample messages into the 'snort' kafka topic and check if you are able to see them in the UI? I would like to see if this works for you, and to ascertain if it is an issue with 'squid' only.
09/09/16-09:09:09.844676 ,1,999158,0,"snort test alert",TCP,192.168.138.158,49188,62.75.195.236,80,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A****,0xF017C4DA,0xABDB8426,,0xF6C9,128,0,2319,40,40960,,,,
Created 12-18-2017 04:14 AM
I tried changing the is_alert field in the template to boolean and deleted and recreated the index. But no luck. I already have alert as a nested field in my template
Created 12-18-2017 10:55 AM
This looks like a defect. I have created METRON-1369 to track this.
Created 05-29-2018 08:52 AM
I am facing the same issue. @jnarayanan have you able to resolve it?