/var/lib/kms-keytrustee/keytrustee/.keytrustee/ is empty

[#1]

I am enabling HDFS data at rest encryption in CDH 6.3 and while adding the KMS service, i noticed that /var/lib/kms-keytrustee/keytrustee/.keytrustee/ is actually doesn't exist on all the KMS hosts when trying to synchronise the KMS hosts private keys. Has anyone come across such issue and what is the way forward for this?

[#2 · Accepted solution]

@RajeshBodolla The general steps which I used to follow for Reinstall KTS from scratch this below, might be you are missing something. 

1. Stop the KMS service
2. Delete the KMS service from the Cloudera Manager UI
3. Then remove all the contents under "rm -rf /var/lib/kms-keytrustee/"

Note - Make sure that the hidden directory is also removed '/var/lib/kms-keytrustee/keytrustee/.keytrustee'

4. Now sync the Active and passive KTS using the following steps:
a. Stop the Key Trustee Server service (Key Trustee Server service > Actions > Stop).
b. Run the following command on the Active Key Trustee Server:
$ sudo rsync -zav --exclude .ssl /var/lib/keytrustee/.keytrustee root@keytrustee02.example.com:/var/lib/keytrustee/.

Note - Replace keytrustee02.example.com with the hostname of the Passive Key Trustee Server.

c. Run the following command on the Passive Key Trustee Server:

$ sudo ktadmin init

d. Start the Key Trustee Server service (Key Trustee Server service > Actions > Start).
e. Enable synchronous replication (Key Trustee Server service > Actions > Setup Enable Synchronous Replication in HA mode).
f. Restart the Key Trustee Server service (Key Trustee Server service > Actions > Restart).

Once this is done, now recreate the KMS service using the steps in the following link

https://www.cloudera.com/documentation/enterprise/latest/topics/key_trustee_kms_ha.html

[#3]

I did follow the similar steps but had the issue. I had to remove all the KTS/KMS installation start from scratch which fixed the issue but this time i added only one server first and then added the other.