Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari infra solr service failing regularly

Ambari infra solr service failing regularly

New Contributor

The Ambari Infra Solr Instance service failing regularly in our hadoop HA cluster.

The service will run successfully for some time after starting it. But it fails again later.

Below are the solr logs.

2019-01-08 06:30:28,057 [coreContainerWorkExecutor-2-thread-1-processing-n:prdhdpmn2.na.ad.example.com:8886_solr] ERROR [   ] org.apache.solr.core.CoreContainer$2 (CoreContainer.java:500) - Error waiting for SolrCore to be created
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [audit_logs_shard0_replica1]
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.solr.core.CoreContainer$2.run(CoreContainer.java:496)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to create core [audit_logs_shard0_replica1]
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:827)
        at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:87)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:467)
        at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:458)
        ... 5 more
Caused by: org.apache.solr.common.SolrException: Could not load conf for core audit_logs_shard0_replica1: Can't load schema managed-schema: [schema.xml] Duplicate field definition for '2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed' [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] and [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]]
        at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:84)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:812)
        ... 8 more
Caused by: org.apache.solr.common.SolrException: Can't load schema managed-schema: [schema.xml] Duplicate field definition for '2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed' [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] and [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]]
        at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:577)
        at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:159)
        at org.apache.solr.schema.ManagedIndexSchema.<init>(ManagedIndexSchema.java:104)
        at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:173)
        at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:47)
        at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:70)
        at org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:108)
        at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:79)
        ... 9 more
Caused by: org.apache.solr.common.SolrException: [schema.xml] Duplicate field definition for '2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed' [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] and [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]]
        at org.apache.solr.schema.IndexSchema.loadFields(IndexSchema.java:642)
        at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:473)
        ... 16 more
2019-01-08 06:30:28,072 [recoveryExecutor-16-thread-1-processing-n:prdhdpmn2.na.ad.example.com:8886_solr x:ranger_audits_shard1_replica1 s:shard1 c:ranger_audits r:core_node1] WARN  [c:ranger_audits s:shard1 r:core_node1 x:ranger_audits_shard1_replica1] org.apache.solr.update.UpdateLog$LogReplayer (UpdateLog.java:1308) - Starting log replay tlog{file=/data/ambari_infra_solr/data/ranger_audits_shard1_replica1/data/tlog/tlog.0000000000001289032 refcount=2} active=false starting pos=0
2019-01-08 06:30:34,982 [commitScheduler-22-thread-1] WARN  [c:ranger_audits s:shard1 r:core_node1 x:ranger_audits_shard1_replica1] org.apache.solr.core.SolrCore (SolrCore.java:1795) - [ranger_audits_shard1_replica1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
2019-01-08 07:52:30,043 [recoveryExecutor-16-thread-1-processing-n:prdhdpmn2.na.ad.example.com:8886_solr x:ranger_audits_shard1_replica1 s:shard1 c:ranger_audits r:core_node1] WARN  [c:ranger_audits s:shard1 r:core_node1 x:ranger_audits_shard1_replica1] org.apache.solr.update.UpdateLog$LogReplayer (UpdateLog.java:1298) - Log replay finished. recoveryInfo=RecoveryInfo{adds=1 deletes=0 deleteByQuery=0 errors=0 positionOfStart=0}


The infra solr service running on Masternode 2.

Looking for your help to fix this issue.

Also, what is the impact if Inforsolr instance service get stopped as we don't see any issues to running jobs.

Don't have an account?
Coming from Hortonworks? Activate your account here