Yeah checkpoints aren't great for that.  I honestly don't rely on them.   There are examples of how to save offsets yourself in  https://github.com/koeninger/kafka-exactly-once/   specifically   https://github.com/koeninger/kafka-exactly-once/blob/master/src/main/scala/example/TransactionalPerPartition.scala   The newer Kafka consumer will allow committing offsets to kafka (which still isn't transactional, but should be better than Zookeeper in most ways).  There's work towards making the new consumer work with spark at   https://issues.apache.org/jira/browse/SPARK-12177   and   https://github.com/koeninger/spark-1/tree/kafka-0.9/external/kafka-beta/src/main/scala/org/apache/spark/streaming/kafka ... View more