KMS Error/Kerberos Issue : Exception while contacting value generator

[#1]

On Hadoop cluster, we see intermitted errors in Cloudera Manager diagnostic logs and it fails job running on them. How to investigate it further? we had this issue few weeks back and it became okay but again started since last two days and its intermittent. 

LoadBalancingKMSClientProvider >>  

KMS provider at  threw an IOException:

java.io.IOException: Exception while contacting value generator 

KMSExceptionsProvider >> 

User:'hdfs/*******' Method:GET URL:https://************.com:16000/kms/v1/key/haasfba/_eek?num_keys=1&eek_op=generate Response:Internal Server Error-java.io.IOException: Exception while contacting value generator

java.io.IOException: java.io.IOException: Exception while contacting value generator

[#2]

Hello @npr20202 ,

Could you specify from which log do you see this error. And what jobs are failing because of this error?

What is the CM version and CDH version?

Please ensure that KMS, Keytrustee Server, and keyHSM are in good health.

Please check KTS log, if the error is there please share the full error stack.

Thanks,

Will

[#3]

@npr20202, Have you resolved your issue? If so, can you please provide the solution here? It will help others who are seeking answers to similar queries.  If you are still experiencing the issue, can you provide the information @willx  has requested?