You can use Nifi to save your Kafka messages into HDFS (for instance). Something like this : - ConsumeKafka : flowfile content is the Kafka message itself, and you have access to some attributes : topic name, partition, offset, key...(but not timestamp !). When i need it I store the timestamp in the key. - ReplaceText : build your backup line using flowfile content and attributes - MergeContent : to build a big file containing multiple Kafka message - Extracttext : to set attribute to be used as filename - PutHDFS : to save the created file into HDFS And you can do the reverse if you need to push it bash to your kafka cluster.
... View more
