Hello @Atradius ,   thank you for reaching out to the Community with your issue of having both NameNodes down.   Do you see in your NN log entries like JvmPauseMonitor saying "Detected pause in JVM or host machine" and a value larger than 1000ms, please? It can be an indication that your service is running out of heap. If it is the NameNode, the short-term solution is to increase the heap and restart the service. A long term solution is to identify why did you run out of heap? E.g. do you face with small files issue? Please read article [1] about how to tackle this.   Losing quorum might be caused by ZK service issue, when the ZK is not in quorum. Please check the ZK logs as well.   Please let us know if you need more input to progress with your investigation.   Best regards: Ferenc   [1] https://blog.cloudera.com/small-files-big-foils-addressing-the-associated-metadata-and-application-challenges/ ... View more

@Sajesh PP Are you able to list the keys using above method?If so, please login and accept the answer. ... View more

We have noticed Ambari infra solr service failing regularly after this. The service running in Masternode2 Below are the errors in solr.log. 2019-01-08 06:30:28,057 [coreContainerWorkExecutor-2-thread-1-processing-n:prdhdpmn2.na.ad.example.com:8886_solr] ERROR [ ] org.apache.solr.core.CoreContainer$2 (CoreContainer.java:500) - Error waiting for SolrCore to be createdjava.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [audit_logs_shard0_replica1] at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.solr.core.CoreContainer$2.run(CoreContainer.java:496) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.solr.common.SolrException: Unable to create core [audit_logs_shard0_replica1] at org.apache.solr.core.CoreContainer.create(CoreContainer.java:827) at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:87) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:467) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:458) ... 5 moreCaused by: org.apache.solr.common.SolrException: Could not load conf for core audit_logs_shard0_replica1: Can't load schema managed-schema: [schema.xml] Duplicate field definition for '2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed' [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] and [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:84) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:812) ... 8 moreCaused by: org.apache.solr.common.SolrException: Can't load schema managed-schema: [schema.xml] Duplicate field definition for '2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed' [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] and [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:577) at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:159) at org.apache.solr.schema.ManagedIndexSchema.<init>(ManagedIndexSchema.java:104) at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:173) at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:47) at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:70) at org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:108) at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:79) ... 9 moreCaused by: org.apache.solr.common.SolrException: [schema.xml] Duplicate field definition for '2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed' [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] and [[[2018-07-03 23:00:00,684 INFO FSNamesystem.audit : allowed{type=boolean,properties=indexed,stored,omitNorms,omitTermFreqAndPositions,sortMissingLast}]]] at org.apache.solr.schema.IndexSchema.loadFields(IndexSchema.java:642) at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:473) ... 16 more2019-01-08 06:30:28,072 [recoveryExecutor-16-thread-1-processing-n:prdhdpmn2.na.ad.example.com:8886_solr x:ranger_audits_shard1_replica1 s:shard1 c:ranger_audits r:core_node1] WARN [c:ranger_audits s:shard1 r:core_node1 x:ranger_audits_shard1_replica1] org.apache.solr.update.UpdateLog$LogReplayer (UpdateLog.java:1308) - Starting log replay tlog{file=/data/ambari_infra_solr/data/ranger_audits_shard1_replica1/data/tlog/tlog.0000000000001289032 refcount=2} active=false starting pos=02019-01-08 06:30:34,982 [commitScheduler-22-thread-1] WARN [c:ranger_audits s:shard1 r:core_node1 x:ranger_audits_shard1_replica1] org.apache.solr.core.SolrCore (SolrCore.java:1795) - [ranger_audits_shard1_replica1] PERFORMANCE WARNING: Overlapping onDeckSearchers=22019-01-08 07:52:30,043 [recoveryExecutor-16-thread-1-processing-n:prdhdpmn2.na.ad.example.com:8886_solr x:ranger_audits_shard1_replica1 s:shard1 c:ranger_audits r:core_node1] WARN [c:ranger_audits s:shard1 r:core_node1 x:ranger_audits_shard1_replica1] org.apache.solr.update.UpdateLog$LogReplayer (UpdateLog.java:1298) - Log replay finished. recoveryInfo=RecoveryInfo{adds=1 deletes=0 deleteByQuery=0 errors=0 positionOfStart=0} ... View more

Thanks Jay 🙂 ... View more