Hey All,
I'm installing a vanilla CM / CDH 6.2.X and running into the issue below from the Navigator Metadata Server:
10:45:54.757 PM INFO JsonModelLoader [main]: Added audit event model for SENTRY 10:45:54.761 PM INFO JsonModelLoader [main]: Added audit event model for SOLR 10:46:10.631 PM WARN SecurityHandler [main]: ServletContext@o.e.j.w.WebAppContext@241e8ea6{/solr,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webapp/],STARTING}{/opt/cloudera/cm/cloudera-navigator-server/wars/solr-4.10.3.war} has uncovered http methods for path: /* 10:46:12.171 PM WARN IndexSchema [coreLoadExecutor-5-thread-1]: Field user is not multivalued and destination for multiple copyFields (2) 10:46:14.883 PM INFO NavServerUtil [main]: Found 4 documents in solr core nav_elements 10:46:14.895 PM INFO NavServerUtil [main]: Found 1 documents in solr core nav_relations 10:46:14.896 PM INFO SolrSchemaUpgrade [main]: Checking if solr schema upgrade is needed. 10:46:14.928 PM INFO SolrSchemaUpgrade [main]: Schema version 2900 already up to date with latest schema 2900. 10:46:14.956 PM INFO SolrSchemaUpgrade [main]: Current database upgrade ordinal is -1, latest upgrade step has ordinal 28 10:46:15.024 PM INFO AbstractSolrUpgrade [main]: Starting upgrade step: com.cloudera.nav.server.upgrade.DeleteEntities : Delete temporary entities. 10:46:15.031 PM INFO NavServerUtil [main]: Found 4 documents in solr core nav_elements 10:46:15.039 PM INFO NavServerUtil [main]: Found 1 documents in solr core nav_relations 10:46:15.042 PM INFO DeleteEntities [main]: Delete temporary relations query: {!join from=identity to=endpoint1Ids fromIndex=nav_elements}fileSystemPath:(*\/_SUCCESS OR *\/.trash OR \/user\/history OR *\/.staging OR \/tmp\/logs) AND sourceType:HDFS 10:46:15.206 PM ERROR SolrCore [qtp1100399456-26]: org.apache.solr.common.SolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker at org.apache.solr.search.CursorMark.<init>(CursorMark.java:104) at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:179) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at com.cloudera.nav.auth.SharedSecretRequestFilter.doFilter(SharedSecretRequestFilter.java:55) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) 10:46:15.221 PM ERROR NavServer [main]: Error upgrading solr configuration or data. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310) at com.cloudera.nav.utils.solr.SolrResultSetIterator.getNextBatch(SolrResultSetIterator.java:96) at com.cloudera.nav.utils.solr.SolrResultSetIterator.hasNext(SolrResultSetIterator.java:76) at com.cloudera.nav.server.upgrade.DeleteEntities.deleteForSide(DeleteEntities.java:248) at com.cloudera.nav.server.upgrade.DeleteEntities.blindDelete(DeleteEntities.java:223) at com.cloudera.nav.server.upgrade.DeleteEntities.upgrade(DeleteEntities.java:76) at com.cloudera.nav.server.upgrade.AbstractSolrUpgrade.upgrade(AbstractSolrUpgrade.java:156) at com.cloudera.nav.server.SolrSchemaUpgrade.performJavaUpgrade(SolrSchemaUpgrade.java:400) at com.cloudera.nav.server.SolrSchemaUpgrade.upgradeSolr(SolrSchemaUpgrade.java:292) at com.cloudera.nav.server.NavServer.run(NavServer.java:434) at com.cloudera.nav.server.NavServer.main(NavServer.java:123) 10:46:15.329 PM INFO NavigatorInfrastructureEventDispatcher [pool-12-thread-1]: Waiting for Solr to Start. 10:46:15.331 PM INFO NavigatorInfrastructureEventDispatcher [pool-12-thread-1]: Succesfully started Solr. 10:46:15.332 PM INFO NavigatorInfrastructureEventDispatcher [pool-12-thread-1]: Waiting for Solr upgrade to complete. 10:46:15.489 PM ERROR NavServerUtil [pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details. 10:46:15.496 PM INFO NavServerUtil [pool-12-thread-1]: Exception occurred when checking for upgrade status. java.lang.RuntimeException: Error encountered in fetching the udate progress information. at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88) at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281) at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231) at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158) at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:23) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10:46:15.506 PM INFO SolrMaintenanceAware [pool-12-thread-1]: Checking for upgrade done status to be set. 10:46:15.514 PM ERROR NavServerUtil [pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details. 10:46:15.514 PM INFO NavServerUtil [pool-12-thread-1]: Exception occurred when checking for upgrade status. java.lang.RuntimeException: Error encountered in fetching the udate progress information. at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88) at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281) at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231) at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158) at com.cloudera.nav.server.SolrMaintenanceAware.pollForUpgradeDone(SolrMaintenanceAware.java:34) at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:24) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10:46:15.516 PM INFO SolrMaintenanceAware [pool-12-thread-1]: Upgrade status not set or available, continue checking for upgrade status to be set.
My backend is a PostgreSQL database. Navigator Metastore Server keeps exiting and restarting as a result:
Thx,
TK
Created 04-28-2019 08:04 AM
Hi
Take a peek in to the below , please try in your lower enviroment and validate before moving to production box.
Created 04-28-2019 09:22 AM
Thank you but I saw that already before posting and tried the solutions from it. However, I don't have these tables in PostgreSQL on CDH / CM 6.2.X:
"DELETE FROM NAV_UPGRADE_ORDINAL;" "INSERT INTO NAV_UPGRADE_ORDINAL VALUES (-1, -1);"
select * from NAV_UPGRADE_ORDINAL;
So I posted since on first impressions, this appears a unique case to 6.2.X and PostgreSQL.
postgres=# \c nav_mws01 Password for user postgres: You are now connected to database "nav_mws01" as user "postgres". nav_mws01=# \dt List of relations Schema | Name | Type | Owner --------+--------------------------------+-------+----------- public | activity_audit | table | nav_mws01 public | activity_last_poll_time | table | nav_mws01 public | audit_schema_version | table | nav_mws01 public | hbase_audit_events | table | nav_mws01 public | hbase_audit_events_2019_04_14 | table | nav_mws01 public | hbase_audit_events_2019_04_15 | table | nav_mws01 public | hbase_audit_events_2019_04_26 | table | nav_mws01 public | hbase_audit_events_2019_04_27 | table | nav_mws01 public | hbase_audit_events_2019_04_28 | table | nav_mws01 public | hdfs_audit_events | table | nav_mws01 public | hdfs_audit_events_2019_04_14 | table | nav_mws01 public | hdfs_audit_events_2019_04_15 | table | nav_mws01 public | hdfs_audit_events_2019_04_26 | table | nav_mws01 public | hdfs_audit_events_2019_04_27 | table | nav_mws01 public | hdfs_audit_events_2019_04_28 | table | nav_mws01 public | hive_audit_events | table | nav_mws01 public | hive_audit_events_2019_04_14 | table | nav_mws01 public | hive_audit_events_2019_04_15 | table | nav_mws01 public | hive_audit_events_2019_04_26 | table | nav_mws01 public | hive_audit_events_2019_04_27 | table | nav_mws01 public | hive_audit_events_2019_04_28 | table | nav_mws01 public | hue_audit_events | table | nav_mws01 public | hue_audit_events_2019_04_14 | table | nav_mws01 public | hue_audit_events_2019_04_15 | table | nav_mws01 public | hue_audit_events_2019_04_26 | table | nav_mws01 public | hue_audit_events_2019_04_27 | table | nav_mws01 public | hue_audit_events_2019_04_28 | table | nav_mws01 public | impala_audit_events | table | nav_mws01 public | impala_audit_events_2019_04_14 | table | nav_mws01 public | impala_audit_events_2019_04_15 | table | nav_mws01 public | impala_audit_events_2019_04_26 | table | nav_mws01 public | impala_audit_events_2019_04_27 | table | nav_mws01 public | impala_audit_events_2019_04_28 | table | nav_mws01 public | nav_count_staging_raw | table | nav_mws01 public | nav_file_info | table | nav_mws01 public | nav_file_info_staging | table | nav_mws01 public | nav_file_reads | table | nav_mws01 public | nav_file_reads_staging | table | nav_mws01 public | nav_file_reads_summary | table | nav_mws01 public | nav_parent_info | table | nav_mws01 public | nav_parent_info_staging | table | nav_mws01 public | nav_service_info | table | nav_mws01 public | nav_service_info_staging | table | nav_mws01 public | navms_audit_events | table | nav_mws01 public | navms_audit_events_2019_04_14 | table | nav_mws01 public | navms_audit_events_2019_04_15 | table | nav_mws01 public | navms_audit_events_2019_04_26 | table | nav_mws01 public | navms_audit_events_2019_04_27 | table | nav_mws01 public | navms_audit_events_2019_04_28 | table | nav_mws01 public | partition_info | table | nav_mws01 public | sentry_audit_events | table | nav_mws01 public | sentry_audit_events_2019_04_14 | table | nav_mws01 public | sentry_audit_events_2019_04_15 | table | nav_mws01 public | sentry_audit_events_2019_04_26 | table | nav_mws01 public | sentry_audit_events_2019_04_27 | table | nav_mws01 public | sentry_audit_events_2019_04_28 | table | nav_mws01 public | solr_audit_events | table | nav_mws01 public | solr_audit_events_2019_04_14 | table | nav_mws01 public | solr_audit_events_2019_04_15 | table | nav_mws01 public | solr_audit_events_2019_04_26 | table | nav_mws01 public | solr_audit_events_2019_04_27 | table | nav_mws01 public | solr_audit_events_2019_04_28 | table | nav_mws01 (62 rows) nav_mws01=# select * from NAV_UPGRADE_ORDINAL; ERROR: relation "nav_upgrade_ordinal" does not exist LINE 1: select * from NAV_UPGRADE_ORDINAL; ^ nav_mws01=#
Thx,
TK