Security and DNS Scanning With Apache NiFi
When I was working in CyberSecurity, I often needed to run a number of command line Linux tools, especially those from Kali. I recommend you take a look at Kali, it is a nice Linux distribution with all the security tools you would like. Please read up on them first as some are dangerous, destructive or in other ways not tools you want to run on an open network or on other people's machines.
My Flow calls a few items and builds up a JSON file to save in Hadoop. I could easily convert this to store in Hive or Phoenix or even send this JSON through Kafka to Storm or Spark for machine learning analysis.
Step 1: GetFile Obtain domains to check (could be JMS, files, file, hard code or fed from an application).
Step 2: ExecuteStreamCommand Call the NS Lookup script to check the TXT record for a domain. TXT records could contain random text or they could have security information from SPF or DKIM. You could also call dig.
Step 3: ExecuteStreamCommand Call the URL Crazy script to check for variations of this domain. This is to check for typosquats and other ways hackers might be messing with similar domains.
Step 4: QueryDNS to query DNS directly with NiFi. You can check out various records including TXT, HINFO, MX, NS and SOA.
Step 5: AttributesToJSON to convert all the attributes we've accumulated and convert this into a single JSON file.
Step 6: PutHDFS store this JSON file forever in Hadoop.
Some enhancement I have considered, adding an UpdateAttribute to change filename and add .json extension. As well as send message to Kafka for Spark processing. And finally upserting into Phoenix for some SQL processing.
I have two shell scripts that I am calling with ExecuteStreamCommand:
/opt/demo/security/url.sh /opt/demo/security/urlcrazy-0.5/urlcrazy -i -f csv -p $@ /opt/demo/security/ns.sh nslookup -type=txt $@ /opt/demo/security/dig.shdig $@ -q hinfo,soa,mx,a,ns
Example JSON Output
hdfs dfs -cat /security/7342161887363724
{"path":"./","execution.command":"/opt/demo/security/url.sh","urlcrazy":"Typo Type,Typo,Valid,Pop,DNS-A,CC-A,Country-A,DNS-MX,Extn\nCharacter Omission,sarkdeveloper.com,true,,,?,,com,\nCharacter Omission,spakdeveloper.com,true,,,?,,com,\nCharacter Omission,spardeveloper.com,true,,,?,,com,\nCharacter Omission,sparkdeeloper.com,true,,,?,,com,\nCharacter Omission,sparkdeveloer.com,true,,,?,,com,\nCharacter Omission,sparkdevelope.com,true,,,?,,com,\nCharacter Omission,sparkdeveloper.cm,true,,,?,,cm,\nCharacter Omission,sparkdeveloper.co,false,,,?,,,\nCharacter Omission,sparkdeveloper.om,false,,,?,,,\nCharacter Omission,sparkdevelopercom,false,,,?,,,\nCharacter Omission,sparkdevelopr.com,true,,,?,,com,\nCharacter Omission,sparkdevelper.com,true,,,?,,com,\nCharacter Omission,sparkdeveoper.com,true,,,?,,com,\nCharacter Omission,sparkdevloper.com,true,,,?,,com,\nCharacter Omission,sparkdveloper.com,true,,,?,,com,\nCharacter Omission,sparkeveloper.com,true,,,?,,com,\nCharacter Omission,sprkdeveloper.com,true,,,?,,com,\nCharacter Repeat,spaarkdeveloper.com,true,,,?,,com,\nCharacter Repeat,sparkddeveloper.com,true,,,?,,com,\nCharacter Repeat,sparkdeeveloper.com,true,,,?,,com,\nCharacter Repeat,sparkdeveeloper.com,true,,,?,,com,\nCharacter Repeat,sparkdevelloper.com,true,,,?,,com,\nCharacter Repeat,sparkdevelooper.com,true,,,?,,com,\nCharacter Repeat,sparkdevelopeer.com,true,,,?,,com,\nCharacter Repeat,sparkdeveloper..com,false,,,?,,com,\nCharacter Repeat,sparkdeveloper.ccom,false,,,?,,,\nCharacter Repeat,sparkdeveloper.comm,false,,,?,,,\nCharacter Repeat,sparkdeveloper.coom,false,,,?,,,\nCharacter Repeat,sparkdeveloperr.com,true,,,?,,com,\nCharacter Repeat,sparkdevelopper.com,true,,,?,,com,\nCharacter Repeat,sparkdevveloper.com,true,,,?,,com,\nCharacter Repeat,sparkkdeveloper.com,true,,,?,,com,\nCharacter Repeat,sparrkdeveloper.com,true,,,?,,com,\nCharacter Repeat,spparkdeveloper.com,true,,,?,,com,\nCharacter Repeat,ssparkdeveloper.com,true,,,?,,com,\nCharacter Swap,psarkdeveloper.com,true,,,?,,com,\nCharacter Swap,saprkdeveloper.com,true,,,?,,com,\nCharacter Swap,spakrdeveloper.com,true,,,?,,com,\nCharacter Swap,spardkeveloper.com,true,,,?,,com,\nCharacter Swap,sparkdeevloper.com,true,,,?,,com,\nCharacter Swap,sparkdeveloepr.com,true,,,?,,com,\nCharacter Swap,sparkdevelope.rcom,false,,,?,,,\nCharacter Swap,sparkdeveloper.cmo,false,,,?,,,\nCharacter Swap,sparkdeveloper.ocm,false,,,?,,,\nCharacter Swap,sparkdeveloperc.om,false,,,?,,,\nCharacter Swap,sparkdevelopre.com,true,,,?,,com,\nCharacter Swap,sparkdevelpoer.com,true,,,?,,com,\nCharacter Swap,sparkdeveolper.com,true,,,?,,com,\nCharacter Swap,sparkdevleoper.com,true,,,?,,com,\nCharacter Swap,sparkdveeloper.com,true,,,?,,com,\nCharacter Swap,sparkedveloper.com,true,,,?,,com,\nCharacter Swap,sprakdeveloper.com,true,,,?,,com,\nCharacter Replacement,aparkdeveloper.com,true,,,?,,com,\nCharacter Replacement,dparkdeveloper.com,true,,,?,,com,\nCharacter Replacement,soarkdeveloper.com,true,,,?,,com,\nCharacter Replacement,spaekdeveloper.com,true,,,?,,com,\nCharacter Replacement,sparjdeveloper.com,true,,,?,,com,\nCharacter Replacement,sparkdebeloper.com,true,,,?,,com,\nCharacter Replacement,sparkdeceloper.com,true,,,?,,com,\nCharacter Replacement,sparkdevekoper.com,true,,,?,,com,\nCharacter Replacement,sparkdeveliper.com,true,,,?,,com,\nCharacter Replacement,sparkdevelooer.com,true,,,?,,com,\nCharacter Replacement,sparkdevelopee.com,true,,,?,,com,\nCharacter Replacement,sparkdeveloper.cim,false,,,?,,,\nCharacter Replacement,sparkdeveloper.con,false,,,?,,,\nCharacter Replacement,sparkdeveloper.cpm,false,,,?,,,\nCharacter Replacement,sparkdeveloper.vom,false,,,?,,,\nCharacter Replacement,sparkdeveloper.xom,false,,,?,,,\nCharacter Replacement,sparkdevelopet.com,true,,,?,,com,\nCharacter Replacement,sparkdeveloprr.com,true,,,?,,com,\nCharacter Replacement,sparkdevelopwr.com,true,,,?,,com,\nCharacter Replacement,sparkdevelpper.com,true,,,?,,com,\nCharacter Replacement,sparkdevrloper.com,true,,,?,,com,\nCharacter Replacement,sparkdevwloper.com,true,,,?,,com,\nCharacter Replacement,sparkdrveloper.com,true,,,?,,com,\nCharacter Replacement,sparkdwveloper.com,true,,,?,,com,\nCharacter Replacement,sparkfeveloper.com,true,,,?,,com,\nCharacter Repla","filename":"7342161887363724","execution.command.args":"sparkdeveloper.com","execution.status":"0","spf":"Server:\t\t10.42.1.20\nAddress:\t10.42.1.20#53\n\nNon-authoritative answer:\nsparkdeveloper.com\ttext = \"v=spf1 ip4:38.113.1.0/24 ip4:38.113.20.0/24 ip4:65.254.224.0/19 ?all\"\n\nAuthoritative answers can be found from:\n\n","execution.error":"","uuid":"f3225b2b-25f3-4e84-b7a6-4dca0c22d362","url":"sparkdeveloper.com","enrich.dns.record0.group0":"\"v=spf1 ip4:38.113.1.0/24 ip4:38.113.20.0/24 ip4:65.254.224.0/19 ?all\""}If you are interested in serious security scanning, check out Metron.
References:
- http://www.openspf.org/SPF_Record_Syntax
- https://www.morningstarsecurity.com/research/urlcrazy
- http://tools.kali.org/
- http://metron.incubator.apache.org/
- yum install bind-utils
- https://en.wikipedia.org/wiki/TXT_record
- http://www.slideshare.net/bunkertor/hadoop-security-54483815
- http://www.dkim.org/
- http://dns-record-viewer.online-domain-tools.com/
- https://en.wikipedia.org/wiki/Typosquatting
