Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

FireEye topology not parsing

Highlighted

FireEye topology not parsing

Explorer

I am trying to parse the following FireEye log-

CEF:0|FireEye|MPS|6.1.0.69991|MC|malware-callback|9|src=195.2.252.157 spt=80 smac=00:0d:66:4d:fc:00 rt=May 08 2016 14:24:45 dst=128.12.95.64 dpt=0 dmac=00:18:74:1c:a1:80 cn1Label=vlan cn1=0 cn2Label=sid cn2=33331600 cs1Label=sname cs1=Trojan.Piptea.2 msg=https://mil.fireeye.com/edp.php?sname\=Trojan.Piptea.2 cs4Label=link cs4=https://172.16.127.7/event_stream/events?event_id\=111 cs5Label=ccName cs5=195.2.252.157 cn3Label=ccPort cn3=80 proto=tcp shost=rescomp-09-149735.Stanford.EDU dvcHost=mslms dvc=172.16.127.7 externalId=111 CEF:0|FireEye|MPS|5.1.0.55701|WI|web-infection|9|src=3.0.0.0 spt=0 smac=00:00:00:00:00:00 dproc=InternetExplorer 6.0 rt=May 05 2016 12:36:22 dst=64.22.138.10 dpt=555 dmac=92:73:75:00:00:35 cs2Label=anomaly cs2=anomaly-tag misc-anomaly cn2Label=sid cn2=0 msg=https://mil.fireeye.com/edp.php?sname\=Exploit.Browser cs4Label=link cs4=https://172.16.127.7/event_stream/events?event_id\=15 fileType=text/html request=vip2.51.la/go.asp?we\=a-free-service-forwebmasters& svid\=22&id\=1153797&tpages\=1&ttimes\=1&tzone\=- 8&tcolor\=24&ssize\=800,600&referrer\=http%3a//88.88 cs1Label=sname cs1=Exploit.Browser shost=web155.discountasp.net dvcHost=mslms dvc=172.16.127.7 externalId=3 CEF:0|FireEye|MPS|6.1.0.69991|MO|malware-object|9|src=195.2.252.153 spt=880 smac=00:0d:66:4d:fc:00 rt=May 10 2016 11:09:31 dst=128.12.95.64 dpt=0 dmac=00:18:74:1c:a1:80 cs2Label=anomaly cs2=anomaly-tag misc-anomaly cn1Label=vlan cn1=0 cn2Label=sid cn2=33331724 cs1Label=sname cs1=Trojan.Piptea.2 msg=https://mil.fireeye.com/edp.php?sname\=Trojan.Piptea.2 cs4Label=link cs4=https://172.16.127.7/event_stream/events?event_id\=254 cs5Label=ccName cs5=ahohonline.com cn3Label=ccPort cn3=80 proto=tcp cs6Label=ccChannel cs6=GET /ufwnltbz/evmhfzlfe.php?id\=1812198572&p\=1 HTTP/1.1::~~User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1)ver52::~~Host: ahohonline.com::~~::~~ shost=rescomp-09-149735.Stanford.EDU dvcHost=mslms dvc=172.16.127.7 externalId=224 CEF:0|FireEye|CMS|7.6.0.334042|WI|web-infection|4|rt=May 25 2016 22:07:50 UTC src=192.168.1.1 dproc=InternetExplorer 7.0 cs3Label=osinfo cs3=Microsoft WindowsXP 32-bit 5.1 sp3 15.0210 filePath=xxx.xx.x.xx:xxxx/metasploit dvchost=axhwmps dvc=192.168.5.6 smac=00:0c:29:d9:2e:e1 cn1Label=vlan cn1=0 externalId=11646 cs4Label=link cs4=https://www.fireeye.com/event_stream/events_for_bot?inc_id\=11646 act=notified cs2Label=anomaly cs2=misc-anomaly cs1Label=sname cs1=Malware.Binary.url

The Exception log from Storm UI is as below -

2016-09-29 06:13:25.115 STDIO [INFO] {"original_string":"CEF:0|FireEye|MPS|6.1.0.69991|MC|malware-callback|9|src=195.2.252.157 spt=80 smac=00:0d:66:4d:fc:00 rt=May 08 2016 14:24:45 dst=128.12.95.64 dpt=0 dmac=00:18:74:1c:a1:80 cn1Label=vlan cn1=0 cn2Label=sid cn2=33331600 cs1Label=sname cs1=Trojan.Piptea.2 msg=https:\/\/mil.fireeye.com\/edp.php?sname\\=Trojan.Piptea.2 cs4Label=link cs4=https:\/\/172.16.127.7\/event_stream\/events?event_id\\=111 cs5Label=ccName cs5=195.2.252.157 cn3Label=ccPort cn3=80 proto=tcp shost=rescomp-09-149735.Stanford.EDU dvcHost=mslms dvc=172.16.127.7 externalId=111 \n \nCEF:0|FireEye|MPS|5.1.0.55701|WI|web-infection|9|src=3.0.0.0 spt=0 smac=00:00:00:00:00:00 dproc=InternetExplorer 6.0 rt=May 05 2016 12:36:22 dst=64.22.138.10 dpt=555 dmac=92:73:75:00:00:35 cs2Label=anomaly cs2=anomaly-tag misc-anomaly cn2Label=sid cn2=0 msg=https:\/\/mil.fireeye.com\/edp.php?sname\\=Exploit.Browser cs4Label=link cs4=https:\/\/172.16.127.7\/event_stream\/events?event_id\\=15 fileType=text\/html request=vip2.51.la\/go.asp?we\\=a-free-service-forwebmasters& svid\\=22&id\\=1153797&tpages\\=1&ttimes\\=1&tzone\\=- 8&tcolor\\=24&ssize\\=800,600&referrer\\=http%3a\/\/88.88 cs1Label=sname cs1=Exploit.Browser shost=web155.discountasp.net dvcHost=mslms dvc=172.16.127.7 externalId=3 \n\nCEF:0|FireEye|MPS|6.1.0.69991|MO|malware-object|9|src=195.2.252.153 spt=880 smac=00:0d:66:4d:fc:00 rt=May 10 2016 11:09:31 dst=128.12.95.64 dpt=0 dmac=00:18:74:1c:a1:80 cs2Label=anomaly cs2=anomaly-tag misc-anomaly cn1Label=vlan cn1=0 cn2Label=sid cn2=33331724 cs1Label=sname cs1=Trojan.Piptea.2 msg=https:\/\/mil.fireeye.com\/edp.php?sname\\=Trojan.Piptea.2 cs4Label=link cs4=https:\/\/172.16.127.7\/event_stream\/events?event_id\\=254 cs5Label=ccName cs5=ahohonline.com cn3Label=ccPort cn3=80 proto=tcp cs6Label=ccChannel cs6=GET \/ufwnltbz\/evmhfzlfe.php?id\\=1812198572&p\\=1 HTTP\/1.1::~~User-Agent: Mozilla\/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1)ver52::~~Host: ahohonline.com::~~::~~ shost=rescomp-09-149735.Stanford.EDU dvcHost=mslms dvc=172.16.127.7 externalId=224  \n\nCEF:0|FireEye|CMS|7.6.0.334042|WI|web-infection|4|rt=May 25 2016 22:07:50 UTC src=192.168.1.1 dproc=InternetExplorer 7.0 cs3Label=osinfo cs3=Microsoft WindowsXP 32-bit 5.1 sp3 15.0210 filePath=xxx.xx.x.xx:xxxx\/metasploit dvchost=axhwmps dvc=192.168.5.6 smac=00:0c:29:d9:2e:e1 cn1Label=vlan cn1=0 externalId=11646 cs4Label=link cs4=https:\/\/www.fireeye.com\/event_stream\/events_for_bot?inc_id\\=11646 act=notified cs2Label=anomaly cs2=misc-anomaly cs1Label=sname cs1=Malware.Binary.url \n"}
2016-09-29 06:13:25.125 o.a.m.p.f.BasicFireEyeParser [WARN] Unable to find timestamp in message: CEF:0|FireEye|MPS|6.1.0.69991|MC|malware-callback|9|src=195.2.252.157 spt=80 smac=00:0d:66:4d:fc:00 rt=May 08 2016 14:24:45 dst=128.12.95.64 dpt=0 dmac=00:18:74:1c:a1:80 cn1Label=vlan cn1=0 cn2Label=sid cn2=33331600 cs1Label=sname cs1=Trojan.Piptea.2 msg=https://mil.fireeye.com/edp.php?sname\=Trojan.Piptea.2 cs4Label=link cs4=https://172.16.127.7/event_stream/events?event_id\=111 cs5Label=ccName cs5=195.2.252.157 cn3Label=ccPort cn3=80 proto=tcp shost=rescomp-09-149735.Stanford.EDU dvcHost=mslms dvc=172.16.127.7 externalId=111 
 
CEF:0|FireEye|MPS|5.1.0.55701|WI|web-infection|9|src=3.0.0.0 spt=0 smac=00:00:00:00:00:00 dproc=InternetExplorer 6.0 rt=May 05 2016 12:36:22 dst=64.22.138.10 dpt=555 dmac=92:73:75:00:00:35 cs2Label=anomaly cs2=anomaly-tag misc-anomaly cn2Label=sid cn2=0 msg=https://mil.fireeye.com/edp.php?sname\=Exploit.Browser cs4Label=link cs4=https://172.16.127.7/event_stream/events?event_id\=15 fileType=text/html request=vip2.51.la/go.asp?we\=a-free-service-forwebmasters& svid\=22&id\=1153797&tpages\=1&ttimes\=1&tzone\=- 8&tcolor\=24&ssize\=800,600&referrer\=http%3a//88.88 cs1Label=sname cs1=Exploit.Browser shost=web155.discountasp.net dvcHost=mslms dvc=172.16.127.7 externalId=3 

CEF:0|FireEye|MPS|6.1.0.69991|MO|malware-object|9|src=195.2.252.153 spt=880 smac=00:0d:66:4d:fc:00 rt=May 10 2016 11:09:31 dst=128.12.95.64 dpt=0 dmac=00:18:74:1c:a1:80 cs2Label=anomaly cs2=anomaly-tag misc-anomaly cn1Label=vlan cn1=0 cn2Label=sid cn2=33331724 cs1Label=sname cs1=Trojan.Piptea.2 msg=https://mil.fireeye.com/edp.php?sname\=Trojan.Piptea.2 cs4Label=link cs4=https://172.16.127.7/event_stream/events?event_id\=254 cs5Label=ccName cs5=ahohonline.com cn3Label=ccPort cn3=80 proto=tcp cs6Label=ccChannel cs6=GET /ufwnltbz/evmhfzlfe.php?id\=1812198572&p\=1 HTTP/1.1::~~User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.1)ver52::~~Host: ahohonline.com::~~::~~ shost=rescomp-09-149735.Stanford.EDU dvcHost=mslms dvc=172.16.127.7 externalId=224  

CEF:0|FireEye|CMS|7.6.0.334042|WI|web-infection|4|rt=May 25 2016 22:07:50 UTC src=192.168.1.1 dproc=InternetExplorer 7.0 cs3Label=osinfo cs3=Microsoft WindowsXP 32-bit 5.1 sp3 15.0210 filePath=xxx.xx.x.xx:xxxx/metasploit dvchost=axhwmps dvc=192.168.5.6 smac=00:0c:29:d9:2e:e1 cn1Label=vlan cn1=0 externalId=11646 cs4Label=link cs4=https://www.fireeye.com/event_stream/events_for_bot?inc_id\=11646 act=notified cs2Label=anomaly cs2=misc-anomaly cs1Label=sname cs1=Malware.Binary.url 

2016-09-29 06:13:25.126 STDIO [ERROR] java.lang.NullPointerException
2016-09-29 06:13:25.126 STDIO [ERROR] at org.apache.metron.parsers.utils.ParserUtils.convertToEpoch(ParserUtils.java:51)
2016-09-29 06:13:25.126 STDIO [ERROR] at org.apache.metron.parsers.fireeye.BasicFireEyeParser.getTimeStamp(BasicFireEyeParser.java:122)
2016-09-29 06:13:25.126 STDIO [ERROR] at org.apache.metron.parsers.fireeye.BasicFireEyeParser.parse(BasicFireEyeParser.java:97)
2016-09-29 06:13:25.126 STDIO [ERROR] at org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:70)
2016-09-29 06:13:25.126 STDIO [ERROR] at backtype.storm.daemon.executor$fn__5495$tuple_action_fn__5497.invoke(executor.clj:670)
2016-09-29 06:13:25.126 STDIO [ERROR] at backtype.storm.daemon.executor$mk_task_receiver$fn__5418.invoke(executor.clj:426) 

The BasicFireEyeParser is not able to find out the Timestamp properly and hence failing to process.

What is the remedy for this problem?

1 REPLY 1
Highlighted

Re: FireEye topology not parsing

Explorer

@Ss i managed to convert timestamp in Kibana in metron, what i did was added below Paser config in metron sensor settings.

PARSER CONFIG

timestampField - timestamp

if you need more details feel free to contact me.

Don't have an account?
Coming from Hortonworks? Activate your account here