Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ranger KMS fails to start after Generating Master Key

Ranger KMS fails to start after Generating Master Key

New Contributor

I installed Ranger on my cluster for the first time ( for the cluster and for me :-)

Using Ambari Version 2.2.1.0 and HDP-2.4.0.0-169 and Postgres as database. 3 nodes on the cluster and separate Ambari server hosting Postgres. All Ranger services installed the same host running NameNode if that matters. All machines are vms running Debian 7 on VMware 6 single physical box.

Ranger service starts and I can access my Service Manager at http://sandbox01.hadoop.private:6080 but when I try starting Ranger KMS service I get the following in the log and the service stops

2016-04-15 11:17:57,792 INFO  RangerKMSDB - Connected to DB : true
2016-04-15 11:17:57,798 INFO  RangerMasterKey - Generating Master Key
2016-04-15 11:17:57,816 INFO  AuditProviderFactory - ==> JVMShutdownHook.run()
2016-04-15 11:17:57,818 INFO  AuditAsyncQueue - Stop called. name=kms.async
2016-04-15 11:17:57,818 INFO  AuditAsyncQueue - Interrupting consumerThread. name=kms.async, consumer=kms.async.multi_dest
2016-04-15 11:17:57,818 INFO  AuditProviderFactory - <== JVMShutdownHook.run()
2016-04-15 11:17:57,818 INFO  AuditAsyncQueue - Caught exception in consumer thread. Shutdown might be in progress
2016-04-15 11:17:57,819 INFO  AuditAsyncQueue - Exiting polling loop. name=kms.async
2016-04-15 11:17:57,819 INFO  AuditAsyncQueue - Calling to stop consumer. name=kms.async, consumer.name=kms.async.multi_dest
2016-04-15 11:17:57,819 INFO  AuditBatchQueue - Stop called. name=kms.async.multi_dest.batch
2016-04-15 11:17:57,819 INFO  AuditBatchQueue - Interrupting consumerThread. name=kms.async.multi_dest.batch, consumer=kms.async.multi_dest.batch.hdfs
2016-04-15 11:17:57,819 INFO  AuditBatchQueue - Stop called. name=kms.async.multi_dest.batch
2016-04-15 11:17:57,819 INFO  AuditBatchQueue - Interrupting consumerThread. name=kms.async.multi_dest.batch, consumer=kms.async.multi_dest.batch.solr
2016-04-15 11:17:57,819 INFO  AuditBatchQueue - Caught exception in consumer thread. Shutdown might be in progress
2016-04-15 11:17:57,819 INFO  AuditAsyncQueue - Exiting consumerThread.run() method. name=kms.async
2016-04-15 11:17:57,819 INFO  AuditBatchQueue - Exiting consumerThread. Queue=kms.async.multi_dest.batch, dest=kms.async.multi_dest.batch.hdfs
2016-04-15 11:17:57,819 INFO  AuditBatchQueue - Calling to stop consumer. name=kms.async.multi_dest.batch, consumer.name=kms.async.multi_dest.batch.hdfs
2016-04-15 11:17:57,820 INFO  AuditBatchQueue - Caught exception in consumer thread. Shutdown might be in progress
2016-04-15 11:17:57,820 INFO  AuditFileSpool - Stop called, queueName=kms.async.multi_dest.batch, consumer=kms.async.multi_dest.batch.hdfs
2016-04-15 11:17:57,821 INFO  AuditBatchQueue - Exiting consumerThread.run() method. name=kms.async.multi_dest.batch
2016-04-15 11:17:57,820 INFO  AuditBatchQueue - Exiting consumerThread. Queue=kms.async.multi_dest.batch, dest=kms.async.multi_dest.batch.solr
2016-04-15 11:17:57,821 INFO  AuditBatchQueue - Calling to stop consumer. name=kms.async.multi_dest.batch, consumer.name=kms.async.multi_dest.batch.solr

Any idea where to go from here ?

Thanks

Greg

4 REPLIES 4

Re: Ranger KMS fails to start after Generating Master Key

@Greg Ladowny

I do not see any error in the logs you mentioned above. Are you able to see any error in logs or ambari while starting the service?

Can you make sure you have enough free memory to start the services.

Re: Ranger KMS fails to start after Generating Master Key

New Contributor

Looks like my second post was treated as a reply to my own question not to yours.

KMS Seems to be complaining about "java.security.InvalidKeyException: Illegal key size" , this would imply that Java Cryptography Extension (JCE) Policy FilesUnlimited strength encryption are missingv, but that apparently was installed by ambari-server setup. Would it not install these on other hosts in the cluster, just on ambari server host ?

Re: Ranger KMS fails to start after Generating Master Key

@Greg Ladowny

Ambari does not setup/ copy JCE policies on all host. Pls read the doc below where it clearly says you need to distribute and install JCE on all nodes -

https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.1.1/bk_Ambari_Security_Guide/content/_installin...

Re: Ranger KMS fails to start after Generating Master Key

New Contributor

The cluster is idle, there is 23G out of 32G free memory on that node at the moment.

I cannot see errors either it just starts when I press start service button in ambari and then silently stops after a minute or so.

/var/log/ranger/kms/catalina.out contains this

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Apr 15, 2016 2:16:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =./webapp, webAppName = /kms
Apr 15, 2016 2:16:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [./webapp] .....
Apr 15, 2016 2:16:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [./webapp].
Apr 15, 2016 2:16:20 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-9292"]
Apr 15, 2016 2:16:20 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat
Apr 15, 2016 2:16:20 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.55
Apr 15, 2016 2:16:20 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
java.security.InvalidKeyException: Illegal key size
        at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039)
        at javax.crypto.Cipher.implInit(Cipher.java:805)
        at javax.crypto.Cipher.chooseProvider(Cipher.java:864)
        at javax.crypto.Cipher.init(Cipher.java:1396)
        at javax.crypto.Cipher.init(Cipher.java:1327)
        at org.apache.hadoop.crypto.key.RangerMasterKey.encryptKey(RangerMasterKey.java:177)
        at org.apache.hadoop.crypto.key.RangerMasterKey.encryptMasterKey(RangerMasterKey.java:153)
        at org.apache.hadoop.crypto.key.RangerMasterKey.generateMasterKey(RangerMasterKey.java:88)
        at org.apache.hadoop.crypto.key.RangerKeyStoreProvider.<init>(RangerKeyStoreProvider.java:91)
        at org.apache.hadoop.crypto.key.RangerKeyStoreProvider$Factory.createProvider(RangerKeyStoreProvider.java:386)
        at org.apache.hadoop.crypto.key.KeyProviderFactory.get(KeyProviderFactory.java:95)
        at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:176)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4992)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5490)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

ERROR: Hadoop KMS could not be started

REASON: java.lang.NullPointerException

Stacktrace:
---------------------------------------------------
java.lang.NullPointerException
        at org.apache.hadoop.crypto.key.kms.server.KMSWebApp.contextInitialized(KMSWebApp.java:178)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4992)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5490)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1575)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1565)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
---------------------------------------------------

Seems to be complaining about "java.security.InvalidKeyException: Illegal key size" , but I have not generated any keys myself, everything was done by ambari. Java was also installed by ambari-server setup, used "[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8"

Don't have an account?
Coming from Hortonworks? Activate your account here