Create ranger_audits collection fails with "Could not initialize class java.time.zone.ZoneRulesProvider"

[#1]

I am trying to re-create a ranger-audits collection for Infra-Solr with numShards=2 instead of 1.  

# curl -k --negotiate -u : "https://fqdn.example.net:8995/solr/admin/collections?action=CREATE&name=ranger_audits&collection.configName=ranger_audits&numShards=2" {
"responseHeader":{
"status":400,
"QTime":1730},
"failure":{
"fqdn.example.net:8995_solr":"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at https://ufqdn.example.net:8995/solr: Error CREATEing SolrCore 'ranger_audits_shard2_replica_n2': Unable to create core [ranger_audits_shard2_replica_n2] Caused by: Could not initialize class java.time.zone.ZoneRulesProvider"},
"success":{
"fqdn.example.net:8995_solr":{
"responseHeader":{
"status":0,
"QTime":1278},
"core":"ranger_audits_shard1_replica_n1"}},
"Operation create caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Underlying core creation failed while creating collection: ranger_audits",
"exception":{
"msg":"Underlying core creation failed while creating collection: ranger_audits",
"rspCode":400},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","org.apache.solr.common.SolrException"],
"msg":"Underlying core creation failed while creating collection: ranger_audits",
"code":400}}

Re-creating the collection with numShards=1 does not create any issues and it is executed correctly.

curl -k --negotiate -u : "https://fqdn.example.com:8995/solr/admin/collections?action=CREATE&name=ranger_audits&collection.configName=ranger_audits&numShards=1"

At the same time, in Solr logs I receive this error:

[c:ranger_audits s:shard2 r:core_node4 x:ranger_audits_shard2_replica_n2] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'ranger_audits_shard2_replica_n2': Unable to create core [ranger_audits_shard2_replica_n2] Caused by: Could not initialize class java.time.zone.ZoneRulesProvider
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1260)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:93)
at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:821)
at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:787)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:538)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:414)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:346)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1630)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:567)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:501)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Unable to create core [ranger_audits_shard2_replica_n2]
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1354)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1221)
... 50 more
Caused by: org.apache.solr.common.SolrException: Could not initialize class java.time.zone.ZoneRulesProvider
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1084)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:905)
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1333)
... 51 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class java.time.zone.ZoneRulesProvider
at java.time.format.DateTimeFormatterBuilder$ZoneTextPrinterParser.getTree(DateTimeFormatterBuilder.java:3726)
at java.time.format.DateTimeFormatterBuilder$ZoneIdPrinterParser.parse(DateTimeFormatterBuilder.java:3856)
at java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.parse(DateTimeFormatterBuilder.java:2209)
at java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.parse(DateTimeFormatterBuilder.java:2209)
at java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.parse(DateTimeFormatterBuilder.java:2209)
at java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.parse(DateTimeFormatterBuilder.java:2219)
at java.time.format.DateTimeFormatter.parseUnresolved0(DateTimeFormatter.java:2010)
at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1939)
at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1819)
at org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.parseInstant(ParseDateFieldUpdateProcessorFactory.java:230)
at org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.validateFormatter(ParseDateFieldUpdateProcessorFactory.java:214)
at org.apache.solr.update.processor.ParseDateFieldUpdateProcessorFactory.init(ParseDateFieldUpdateProcessorFactory.java:189)
at org.apache.solr.core.PluginBag.initInstance(PluginBag.java:106)
at org.apache.solr.core.PluginBag.createPlugin(PluginBag.java:148)
at org.apache.solr.core.PluginBag.init(PluginBag.java:290)
at org.apache.solr.core.PluginBag.init(PluginBag.java:279)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1008)
... 53 more

Cluster is kerberized and cloudera runtime 7.1.6-1.

Can anyone can give me a hint on how to fix this issue?

[#2 · Accepted solution]

Hello @GregDol 

Thanks for using Cloudera Community. This is an Old Post, yet I am sharing a review for other Community Users, in case they encounter the concerned issue. 

Reviewing the Post indicates the Issue is likely caused by missing "rt.jar" in the JDK used by the Solr Service. In other words, the concerned Class "java.time.zone.ZoneRulesProvider" is present in "/usr/java/jdk1.8.0_232-cloudera/jre/lib/rt.jar" for CDP Cluster (Path & Version may be different for Customer). The concerned JDK is used by the various Services running including Solr.

In your scenario, the concerned Class appears to be missing. Assuming your team wish to confirm whether a JAR consists of the Class, you may use Command [

/usr/java/jdk1.8.0_232-cloudera/bin/jar -tvf /usr/java/jdk1.8.0_232-cloudera/jre/lib/rt.jar] to list the Classes, which should include "java.time.zone.ZoneRulesProvider" as well. If your Team is using any different JDK, Review the same & ensure the "rt.jar" is present for Class "java.time.zone.ZoneRulesProvider" to be loaded successfully. 

Regards, Smarak