kafka Data encryption at rest

[#1]

I am looking for best practice with Kafka data encryption at rest. I am aware of custom solutions using AES for producer and consumer; however, looking for alternatives or best practice.

[#2]

@schintalapani

[#3 · Accepted solution]

we could in effect write a consumer and producer that can watch the folder directory on HDFS using maybe INotifiy and then pull that file off, if they have permissions it should decrypt while its collecting and then write the blob into Kafka. Then the Consumer can read from Kafka and write into HDFS again with TDE. There are more discussion here .

http://www.confluent.io/blog/apache-kafka-0.9-is-released

[#4]

Hi @Sunile Manjee , I am also looking for Encryption at rest for Kafka messages. But prefer not to pass on the messages to the HDFS. Consider our requirement more or less like "disk theft" which contains credit card transaction logs for 100s of credit card holders or military data.

Thanks in advance.

[#5]

An example of using RSA+AES: http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-proof-concept