Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

Highlighted

Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

New Contributor

I am having a problem with Hadoop 2.4 stack on CentOS 6.8 final with OpenJDK 1.8.0 where the Ranger KMS service will not stay running. The service crashes after a few moments and the Ambari interface shows an error.

The error regarding illegal key size immediately made me think of Sun/Oracle Java and needing to deploy the Java JCE policy files. However, I am not using Sun/Oracle Java and therefore should not require the JCE policy files as those are allegedly built in with OpenJDK. Any idea what could be causing this?

Output of command 'java -version'

openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

I am getting the following exception in the logs:

catalina.out

Jun 02, 2016 11:01:44 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
java.security.InvalidKeyException: Illegal key size
        at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039)
        at javax.crypto.Cipher.implInit(Cipher.java:805)
        at javax.crypto.Cipher.chooseProvider(Cipher.java:864)
        at javax.crypto.Cipher.init(Cipher.java:1396)
        at javax.crypto.Cipher.init(Cipher.java:1327)
        at org.apache.hadoop.crypto.key.RangerMasterKey.encryptKey(RangerMasterKey.java:177)
        at org.apache.hadoop.crypto.key.RangerMasterKey.encryptMasterKey(RangerMasterKey.java:153)
        at org.apache.hadoop.crypto.key.RangerMasterKey.generateMasterKey(RangerMasterKey.java:88)
        at org.apache.hadoop.crypto.key.RangerKeyStoreProvider.<init>(RangerKeyStoreProvider.java:91)
        at org.apache.hadoop.crypto.key.RangerKeyStoreProvider$Factory.createProvider(RangerKeyStoreProvider.java:386)
        at org.apache.hadoop.crypto.key.KeyProviderFactory.get(KeyProviderFactory.java:95)
        at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:176)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1572)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1562)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


ERROR: Hadoop KMS could not be started


REASON: java.lang.NullPointerException


Stacktrace:
---------------------------------------------------
java.lang.NullPointerException
        at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:178)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1572)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1562)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
---------------------------------------------------


kms.log

...
2016-06-02 23:01:46,278 DEBUG RangerPolicyRepository - policy evaluation order: 1 policies
2016-06-02 23:01:46,278 DEBUG RangerPolicyRepository - policy evaluation order: #1 - policy id=1; name=HWHDP_POC_kms-1-20160602203540; evalOrder=9974
2016-06-02 23:01:46,278 DEBUG init - [PERF] RangerPolicyEngine.init(hashCode=4f531daa): 21
2016-06-02 23:01:46,278 DEBUG RangerPolicyEngineImpl - <== RangerPolicyEngineImpl()
2016-06-02 23:01:46,278 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=HWHDP_POC_kms).loadPolicy()
2016-06-02 23:01:46,280 DEBUG RangerKmsAuthorizer - <== RangerkmsAuthorizer.init()
2016-06-02 23:01:46,281 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=HWHDP_POC_kms).run()		
2016-06-02 23:01:46,287 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=HWHDP_POC_kms).loadPolicy()
2016-06-02 23:01:46,287 DEBUG PolicyRefresher - ==> PolicyRefresher(serviceName=HWHDP_POC_kms).loadPolicyfromPolicyAdmin()
2016-06-02 23:01:46,287 DEBUG RangerAdminRESTClient - ==> RangerAdminRESTClient.getServicePoliciesIfUpdated(1)
2016-06-02 23:01:46,300 DEBUG RangerAdminRESTClient - <== RangerAdminRESTClient.getServicePoliciesIfUpdated(1): null
2016-06-02 23:01:46,300 DEBUG PolicyRefresher - PolicyRefresher(serviceName=HWHDP_POC_kms).run(): no update found. lastKnownVersion=1
2016-06-02 23:01:46,300 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=HWHDP_POC_kms).loadPolicyfromPolicyAdmin()
2016-06-02 23:01:46,300 DEBUG init - [PERF] PolicyRefresher.loadPolicy(serviceName=HWHDP_POC_kms): 13
2016-06-02 23:01:46,301 DEBUG PolicyRefresher - <== PolicyRefresher(serviceName=HWHDP_POC_kms).loadPolicy()
2016-06-02 23:01:46,319 INFO  log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-06-02 23:01:46,319 INFO  log - ------------------ Ranger KMSWEbApp---------------------
2016-06-02 23:01:46,319 INFO  log - provider string = dbks://http@localhost:9292/kms
2016-06-02 23:01:46,319 INFO  log - URI = dbks://http@localhost:9292/kms scheme = dbks
2016-06-02 23:01:46,319 INFO  log - kmsconf size= 194 kms classname=org.apache.hadoop.conf.Configuration
2016-06-02 23:01:46,319 INFO  log - ----------------INstantiating key provider ---------------
2016-06-02 23:01:48,093 INFO  RangerKMSDB - Connected to DB : true
2016-06-02 23:01:48,098 INFO  RangerMasterKey - Generating Master Key
2016-06-02 23:01:48,129 INFO  AuditProviderFactory - ==> JVMShutdownHook.run()
2016-06-02 23:01:48,129 INFO  AuditProviderFactory - JVMShutdownHook: Signalling async audit cleanup to start.
2016-06-02 23:01:48,145 INFO  AuditProviderFactory - RangerAsyncAuditCleanup: Starting cleanup
2016-06-02 23:01:48,145 INFO  AuditProviderFactory - JVMShutdownHook: Waiting up to 30 seconds for audit cleanup to finish.
2016-06-02 23:01:48,146 INFO  AuditAsyncQueue - Stop called. name=kms.async
2016-06-02 23:01:48,146 INFO  AuditAsyncQueue - Interrupting consumerThread. name=kms.async, consumer=kms.async.multi_dest
2016-06-02 23:01:48,146 INFO  AuditProviderFactory - RangerAsyncAuditCleanup: Done cleanup
2016-06-02 23:01:48,146 INFO  AuditProviderFactory - RangerAsyncAuditCleanup: Waiting to audit cleanup start signal
2016-06-02 23:01:48,146 INFO  AuditProviderFactory - JVMShutdownHook: Audit cleanup finished after 1 milli seconds
2016-06-02 23:01:48,146 INFO  AuditProviderFactory - JVMShutdownHook: Interrupting ranger async audit cleanup thread
2016-06-02 23:01:48,146 INFO  AuditProviderFactory - <== JVMShutdownHook.run()
2016-06-02 23:01:48,146 INFO  AuditAsyncQueue - Caught exception in consumer thread. Shutdown might be in progress
2016-06-02 23:01:48,147 INFO  AuditAsyncQueue - Exiting polling loop. name=kms.async
2016-06-02 23:01:48,147 INFO  AuditAsyncQueue - Calling to stop consumer. name=kms.async, consumer.name=kms.async.multi_dest
2016-06-02 23:01:48,147 INFO  AuditBatchQueue - Stop called. name=kms.async.multi_dest.batch
2016-06-02 23:01:48,147 INFO  AuditBatchQueue - Interrupting consumerThread. name=kms.async.multi_dest.batch, consumer=kms.async.multi_dest.batch.hdfs
2016-06-02 23:01:48,147 INFO  AuditBatchQueue - Stop called. name=kms.async.multi_dest.batch
2016-06-02 23:01:48,147 INFO  AuditBatchQueue - Interrupting consumerThread. name=kms.async.multi_dest.batch, consumer=kms.async.multi_dest.batch.solr
2016-06-02 23:01:48,147 INFO  AuditAsyncQueue - Exiting consumerThread.run() method. name=kms.async
2016-06-02 23:01:48,147 INFO  AuditBatchQueue - Caught exception in consumer thread. Shutdown might be in progress
12 REPLIES 12

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

Expert Contributor

@Chris Twilleager You need to have the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File on all hosts in the cluster.

Please refer this for installing

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_KMS_Admin_Guide/content/ch_instal...

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

New Contributor

Thank you for the response. However, you are incorrect in your suggestion as your solution only applies to people who are using Sun/Oracle Java. I do not need to install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy because I am using OpenJDK.

Outuput of 'java -version' on all hosts:

openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

Per the documentation for installing Ranger KMS:

"If you use the OpenJDK package, the JCE file is already built into the package."

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

Expert Contributor
@Chris Twilleager

Please confirm that Ranger KMS is using the Java OpenJDK. But error definitely looks like the JCE issue

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

New Contributor

OpenJDK is the only Java package installed on the hosts. I just went back through the entire cluster and verified that no other Java installation is present. As OpenJDK is the only Java available, what else could KMS use besides that?

I went through the advanced properties for RangerKMS, but did not find anywhere to specify the use of an explicit version of Java.

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

You can check to see by evaluating this code on the machine and checking the return value -- 128 means the cryptographic strength is limited, Integer.MAX_VALUE (2^31 - 1) means it is not.
Cipher.getMaxAllowedKeyLength("AES")

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

New Contributor

@Andy LoPresto I believe I did something similar to what you're asking. I found this bit of code:

https://gist.github.com/jehrhardt/5167854

import javax.crypto.Cipher;
import java.security.NoSuchAlgorithmException;

public class KeyLengthDetector {
  public static void main(String[] args) {
    int allowedKeyLength = 0;

    try {
      allowedKeyLength = Cipher.getMaxAllowedKeyLength("AES");
    } catch (NoSuchAlgorithmException e) {
      e.printStackTrace();
    }

    System.out.println("The allowed key length for AES is: " + allowedKeyLength);
  }
}

I compiled and ran it on the edge node that Ranger KMS is running on. Output is:

The allowed key length for AES is: 2147483647

It appears as though I DO actually have unlimited strength security with OpenJDK. I am at a total loss.

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

New Contributor

To update this issue, I went ahead and replaced the entire cluster's running edition of OpenJDK 8 with Oracle JDK + JCE. JCE files have been verified as being present on ALL nodes in the cluster via the following code:

import javax.crypto.Cipher;
import java.security.NoSuchAlgorithmException;

public class KeyLengthDetector {
  public static void main(String[] args) {
    int allowedKeyLength = 0;

    try {
      allowedKeyLength = Cipher.getMaxAllowedKeyLength("AES");
    } catch (NoSuchAlgorithmException e) {
      e.printStackTrace();
    }

    System.out.println("The allowed key length for AES is: " + allowedKeyLength);
  }
}

The output of the above Java code is:

The allowed key length for AES is: 2147483647

Java's home directory ($JAVA_HOME) is set for all users via /etc/profile to /usr/java/default. I should also point out that all other services in the cluster are running fine. Only Ranger KMS is not working.

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

Do you still have this issue?

Re: Ranger KMS crashes with Illegal Key Size exception in logs - OpenJDK

New Contributor

Yes, I still have this problem.

Don't have an account?
Coming from Hortonworks? Activate your account here