Navigator metadata server keep on crashing after changing to Mysql DB

[#1]

Hi there,

I have enabled Navigator metadata server to use mysql db, but it keep on crashing with solr error, hope someone can help on this.

Thanks

SolrCore

[qtp2123908275-75]: org.apache.solr.common.SolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker at org.apache.solr.search.CursorMark.<init>(CursorMark.java:104) at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:179) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:256) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at com.cloudera.nav.auth.SharedSecretRequestFilter.doFilter(SharedSecretRequestFilter.java:55) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745)

[#2]

Hi, can you find a solution for this issue?

Thanks

[#3]

@elifsurmeli,

This usually indicates that the Solr data is out of sync (version-wise) with what nav_upgrade_ordinal table in your Navigator Metadata Server database.

Please do the following to see if that is the cause:

(1)  on the Navigator Metadata host, run rpm -qa |grep cloudera and post the results (so we can confirm the version of Navigator

(2)  Run the following query in your Navigator Metadata Server database (via your favorite db tool):

select * from NAV_UPGRADE_ORDINAL;

NOTE:  The above assumes that you are "using" the Navigator Metadata Server database in your database server client.

With that information, we can see if they may be out of sync and we can decide if we need to adjust the values.

[#4]

Hi @bgooley ,

(1)rpm -qa | grep cloudera gives the following output:

cloudera-manager-agent-5.13.1-1.cm5131.p0.2.el7.x86_64
cloudera-manager-daemons-5.13.1-1.cm5131.p0.2.el7.x86_64
cloudera-manager-server-5.13.1-1.cm5131.p0.2.el7.x86_64

(2)And the query result is:

ordinal | old_ordinal
---------+-------------
       26 |             26

Thanks

[#5]

So... what happened next? I also have this issue, and you leave me hanging!

[#6 · Accepted solution]

I had to reset the Navigator metadata server and followed the steps below:

1) Stop Navigator Metadata Server from CM

2) Take a back up of Navigator Metadata Server Storage Dir (if you want to archive the old data)

3) mv <storage_dir> < new_location>

4) Back up your navms database

5) In NMS database run:

"DELETE FROM NAV_UPGRADE_ORDINAL;" "INSERT INTO NAV_UPGRADE_ORDINAL VALUES (-1, -1);"

6) Start Navigator Metadata Server

this will delete all metadata, custom data added in the Navigator UI but does not affect the Audit logs visible under Navigator UI -> Audits.

[#7]

Sadly this doesn't work for me- my Navigator DB doesn't have this table.

But I'm not upgrading - I'm using oracle in the first place.

Maybe it's another kind of installation error?

Or do I need a software upgrade?

$ rpm -qa | grep cloudera
cloudera-manager-agent-5.12.1-1.cm5121.p0.6.el6.x86_64
cloudera-manager-daemons-5.12.1-1.cm5121.p0.6.el6.x86_64
cloudera-manager-server-5.12.1-1.cm5121.p0.6.el6.x86_64

3:33:19.872 PM ERROR SolrCore
[qtp1384454980-63]: org.apache.solr.common.SolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker
at org.apache.solr.search.CursorMark.<init>(CursorMark.java:104)