Created on 12-18-2016 03:00 AM
A web crawler bot written in Spark with Kafka and Tika to replace Nutch. It renders Javascript and processes files with Tika.
https://github.com/uscdataScience/sparkler/wiki/sparkler-0.1