Navigator Meta server not coming up

[#1]

After restarting CMS , navigator metadata server is not comign up and i see these errors in the logs.

Anyone has any pointers on how to get past this issue ? is this a bug ?

Oct 25, 9:18:04.061 AM WARN org.apache.solr.schema.IndexSchema
[coreLoadExecutor-5-thread-1]: Field user is not multivalued and destination for multiple copyFields (2)
Oct 25, 9:18:04.567 AM ERROR org.apache.solr.core.CoreContainer
[coreLoadExecutor-5-thread-1]: Error creating core [nav_elements]: Error opening new searcher
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:881)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:654)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1574)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1686)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:853)
... 8 more
Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/dcf/hdp/cloudera-scm-navigator/solr/nav_elements/data/index/_bjz6.fnm")
at org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
at org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:134)
at org.apache.lucene.codecs.lucene46.Lucene46FieldInfosReader.read(Lucene46FieldInfosReader.java:57)
at org.apache.lucene.index.SegmentReader.readFieldInfos(SegmentReader.java:289)
at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:864)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:816)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1537)
... 10 more

[#2]

I'm having the same issue, has anyone found a fix for this yet?

Thanks

Chris

[#3]

What i did was i backed up this directory and restarted the metadataserver  as a workaround. it wil rebuild all indexes from scratch. Just verify your audits after restart. 

sudo mv /dcf/hdp/cloudera-scm-navigator /dcf/hdp/cloudera-scm-navigator.bkp 

Restart Navigator 

[#4]

Hi Desind,

Thanks for the quick reply and solution, my paths are different to the ones you describe so i did the following:

Stop Navigator

mv /var/lib/cloudera-scm-navigator /var/lib/cloudera-scm-navigator.bak

Start Navigator

The /var/lib/cloudera-scm-navigator was regnerated when Navigator was started but the following folfer was not regenerated: /var/lib/cloudera-scm-navigator/diagnosticData

The Navigator Metadata Server is working ok now but just wondered why this diagnosticData folder is missing.

Many Thanks,

Chris

[#5]

Hi

noticing this error 

"Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/var/lib/you_metadata_path")"

When errors like this appear in the logs its usually a sign that the internal solr core index has become corrupted somehow from missing files, and the index will need to be regenerated again over time as fresh metadata comes in.

Stopping the Navigator Metadat Server, Clearing the metadata directory and starting a fresh index like this is the best option if this occurs.. It seems to occur most commonly from disks filling up, or a hard server crash of some sort.

-T