I would suggest you use HDFS connect rather than Spark Streaming as it is more fault tolerant. Kafka Connect is built into the base Kafka libraries, but you need to compile and add HDFS Connect separately to the classpath of Connect. Build from here: https://github.com/confluentinc/kafka-connect-hdfs and use a tagged branch rather than master as the releases are publicly available libraries, not SNAPSHOT builds that require you to compile Kafka from source.
... View more
