How do you achieve high availability in HDFS when Ranger KMS is down or the metastore is down?

hduraiswamy — Tue, 18 Oct 2016 22:42:00 GMT

Re: How do you achieve high availability in HDFS when Ranger KMS is down or the metastore is down?

amcbarnett — Tue, 18 Oct 2016 22:57:44 GMT

Scenario 1: Ranger KMS DB is down but Node is Up

The keys are cached for a time. You can still read the data in the encrypted folder. HDFS has knowledge of the encryption zone key
I assume that The Ranger KMS Service is still up, while the DB/ metastore is down.
If you know the database cannot be recovered, and you don¹t have a back up of the keystore, you immediately begin to remove the encryption zone.
You log in as an authorized user, or hdfs and begin copying the files to an unencrypted area and then remove the encrypted zone.
I just tested this on my cluster

Scenario 2: The entire node was down. This means BOTH the Ranger DB and the Ranger KMS Service is down.

The Encryption Zone key is the Ranger KMS DB (Metastore) and you can also export and save to a file.
You should back up and also make the Ranger KMS DB highly available.
Once you export to a keystore file, you back up the file.
If the cluster node goes down, you restore the Ranger KMS DB again from backup.
If you cannot restore Ranger KMS DB from back up, you create a completely new Ranger KMS Db and get the backup Keystore file and as a special user run a script to import the key back to the newly created database.
You can associate once again the encryption zone folder with the key using HDFS commands.
If you Don¹t have BOTH the Keystore file and the Ranger KMS DB to restore then you don¹t have any option. The file remains encrypted.

See this article for script to export and import keys:

hduraiswamy — Tue, 18 Oct 2016 23:01:07 GMT

Thanks my friend!