metron alerts UI is empty

[#1]

I have ingested squid telemtery into metron and can see alerts in the kibana dashboard (I have injested threat intel and setup risk scoring). However the metron alerts UI is empty. Do I need additional configuration for these alerts to show up in the alerts UI?

[#2]

Only 'alerts' will appear in the Alerts UI. So what is an alert then, you ask?

Well, not all telemetry in Metron is treated as an alert. Only telemetry that is specifically marked with a field { "is_alert": "true" } is treated as an alert. This gives the user the flexibility to define which telemetry will go through additional threat triage processing.

In your case, the Squid telemetry does not have this field and so is not treated as an alert. For testing purposes, you can add this field to your Squid telemetry by creating a simple enrichment that adds the field "is_alert" and sets it to "true".

Hope this makes sense.

[#3]

The squid telemetry does have is_alert true.that is the first thing I checked. I have done threat intel enrichment and is_alert is set to true based on the threat intel

[#4]

Do you see anything in the Alerts UI? Or is just this specific Squid data that is missing? What version of Metron are you running? How did you deploy Metron?

[#5]

Nothing at all in the alerts UI. I see 5 alerts in kibanan and 12 non alert events. But alerts UI is empty. I am running 1.3.1.0-37 installed through the ambari mpack

[#6]

hcp 1.3.1.0-37

[#7]

I've been facing same issue, metron-alarms-ui wasn't receiving any alarms, after applying changes to template and adding field "alert", type nested, metron-alarms-ui started receiving some messages(little bit messed up, which is next thing to understand), but looking at your template, i notice that field is_alert is type string, while in my schema it is boolean, maybe this makes some problems.

[#8]

I am running 1.3.1.0-37. Deployed it through the ambari mpack

[#9]

@jnarayanan

You might be running into METRON-1283, which is fixed in more recent versions of HCP. You can confirm as follows - Go to http://<es-master-node>:9200/squid*/_mappings, and look for an "alerts" field. If you are missing this field from the mapping, then it is likely the reason for Alerts UI to be empty.

In order to fix the issue, follow these steps:

* Clear all existing Elasticsearch indices

* Go to Ambari UI -> Services -> Metron -> 'Service Actions' dropdown -> Elasticsearch Template Install

* Re-ingest data into 'squid' telemetry.

And you should now be able to see entries in the Alerts UI.

[#10]

I tried re-installing the elastic template. Still no luck

[#11]

Can you paste the output when you open 'http://<es-master-node>:9200/squid*/_mappings' ?

[#12]

properties
action
type	"string"
index	"not_analyzed"
adapter:geoadapter:begin:ts
type	"string"
adapter:geoadapter:end:ts
type	"string"
adapter:stellaradapter:begin:ts
type	"string"
adapter:stellaradapter:end:ts
type	"string"
bytes
type	"integer"
code
type	"string"
index	"not_analyzed"
domain_without_subdomains
type	"string"
index	"not_analyzed"
elapsed
type	"integer"
enrichmentjoinbolt:joiner:ts
type	"string"
enrichments:geo:ip_dst_addr:city
type	"string"
enrichments:geo:ip_dst_addr:country
type	"string"
enrichments:geo:ip_dst_addr:dmaCode
type	"string"
enrichments:geo:ip_dst_addr:latitude
type	"string"
enrichments:geo:ip_dst_addr:locID
type	"string"
enrichments:geo:ip_dst_addr:location_point
type	"string"
enrichments:geo:ip_dst_addr:longitude
type	"string"
enrichments:geo:ip_dst_addr:postalCode
type	"string"
enrichmentsplitterbolt:splitter:begin:ts
type	"string"
enrichmentsplitterbolt:splitter:end:ts
type	"string"
extra
type	"string"
full_hostname
type	"string"
index	"not_analyzed"
guid
type	"string"
ip_dst_addr
type	"ip"
ip_dst_port
type	"integer"
ip_src_addr
type	"ip"
ip_src_port
type	"integer"
is_malicious
type	"string"
method
type	"string"
index	"not_analyzed"
original_string
type	"string"
source:type
type	"string"
index	"not_analyzed"
threatinteljoinbolt:joiner:ts
type	"string"
threatintelsplitterbolt:splitter:begin:ts
type	"string"
threatintelsplitterbolt:splitter:end:ts
type	"string"
timestamp
type	"date"
format	"epoch_millis"
url
type	"string"

[#13]

{"squid_index_2017.12.14.09":{"mappings":{"squid_doc":{"_timestamp":{"enabled":true},"properties":{"action":{"type":"string","index":"not_analyzed"},"adapter:simplehbaseadapter:begin:ts":{"type":"string"},"adapter:simplehbaseadapter:end:ts":{"type":"string"},"adapter:threatinteladapter:begin:ts":{"type":"string"},"adapter:threatinteladapter:end:ts":{"type":"string"},"alert":{"type":"nested"},"bytes":{"type":"integer"},"code":{"type":"string","index":"not_analyzed"},"domain_without_subdomains":{"type":"string","index":"not_analyzed"},"elapsed":{"type":"integer"},"enrichmentjoinbolt:joiner:ts":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:domain":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:domain_created_timestamp":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:home_country":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:owner":{"type":"string"},"enrichments:hbaseEnrichment:domain_without_subdomains:whois:registrar":{"type":"string"},"enrichmentsplitterbolt:splitter:begin:ts":{"type":"string"},"enrichmentsplitterbolt:splitter:end:ts":{"type":"string"},"full_hostname":{"type":"string","index":"not_analyzed"},"guid":{"type":"string"},"ip_dst_addr":{"type":"string","index":"not_analyzed"},"ip_src_addr":{"type":"string"},"is_alert":{"type":"string"},"method":{"type":"string","index":"not_analyzed"},"original_string":{"type":"string"},"source:type":{"type":"string","index":"not_analyzed"},"threat:triage:rules:0:comment":{"type":"string"},"threat:triage:rules:0:name":{"type":"string"},"threat:triage:rules:0:reason":{"type":"string"},"threat:triage:rules:0:score":{"type":"long"},"threat:triage:rules:1:comment":{"type":"string"},"threat:triage:rules:1:name":{"type":"string"},"threat:triage:rules:1:reason":{"type":"string"},"threat:triage:rules:1:score":{"type":"long"},"threat:triage:score":{"type":"double"},"threatinteljoinbolt:joiner:ts":{"type":"string"},"threatintels:hbaseThreatIntel:domain_without_subdomains:zeusList":{"type":"string"},"threatintelsplitterbolt:splitter:begin:ts":{"type":"string"},"threatintelsplitterbolt:splitter:end:ts":{"type":"string"},"timestamp":{"type":"date","format":"epoch_millis"},"url":{"type":"string"}}}}}}

[#14]

Hmm... this seems about right to me. It should be working, in principle.

In my environment, I have seen snort alerts working readily. Can you try ingesting the following sample messages into the 'snort' kafka topic and check if you are able to see them in the UI? I would like to see if this works for you, and to ascertain if it is an issue with 'squid' only.

09/09/16-09:09:09.844676 ,1,999158,0,"snort test alert",TCP,192.168.138.158,49188,62.75.195.236,80,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A****,0xF017C4DA,0xABDB8426,,0xF6C9,128,0,2319,40,40960,,,,

[#15]

I tried changing the is_alert field in the template to boolean and deleted and recreated the index. But no luck. I already have alert as a nested field in my template

[#16]

This looks like a defect. I have created METRON-1369 to track this.

[#17]

I am facing the same issue. @jnarayanan have you able to resolve it?