Spark Program throwing error java.net.UnknownHostException

[#1]

Hi ,

We have enabled KMS in our server, after enabling KMS and encryption , we are getting following error randomly , Is there anything we can try to resolve this issue ?

org.apache.spark.SparkException: Job aborted due to stage failure: Task 58 in stage 12.0 failed 4 times, most recent failure: Lost task 58.3 in stage 12.0 (TID 4151, lpdn0307.): java.io.IOException: java.lang.IllegalArgumentException: java.net.UnknownHostException: edhpen1262 at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:572) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:850) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:209) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:205) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:94) at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:205) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388) at org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1440) at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1510)

We verified the DNS resolution and hostname , everything is looking good.

What else may be the issue.

Thanks,

Vishnu Ravi

[#2]

Hello, 

Can you provide what versions of CDH/CM you are currently running? Have any other services/componets been impacted after enabling KMS? 

Thanks, 

Jordan 

[#3]

Did anyone found the solution for this issue? I have a similar issue, We are using CM5.11 and installed Spark2 seperately on other server. When we enable KMS on the cluster then Spark2 throws Unknown Host exception.