Reply
Highlighted
Explorer
Posts: 18
Registered: ‎01-27-2019

CM / CDH 6.2.X w/ PSQL:Cursor functionality requires a sort containing a uniqueKey field tie breaker

[ Edited ]

Hey All,

 

I'm installing a vanilla CM / CDH 6.2.X and running into the issue below from the Navigator Metadata Server:

 

10:45:54.757 PM	INFO	JsonModelLoader	
[main]: Added audit event model for SENTRY
10:45:54.761 PM	INFO	JsonModelLoader	
[main]: Added audit event model for SOLR
10:46:10.631 PM	WARN	SecurityHandler	
[main]: ServletContext@o.e.j.w.WebAppContext@241e8ea6{/solr,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webapp/],STARTING}{/opt/cloudera/cm/cloudera-navigator-server/wars/solr-4.10.3.war} has uncovered http methods for path: /*
10:46:12.171 PM	WARN	IndexSchema	
[coreLoadExecutor-5-thread-1]: Field user is not multivalued and destination for multiple copyFields (2)
10:46:14.883 PM	INFO	NavServerUtil	
[main]: Found 4 documents in solr core nav_elements
10:46:14.895 PM	INFO	NavServerUtil	
[main]: Found 1 documents in solr core nav_relations
10:46:14.896 PM	INFO	SolrSchemaUpgrade	
[main]: Checking if solr schema upgrade is needed.
10:46:14.928 PM	INFO	SolrSchemaUpgrade	
[main]: Schema version 2900 already up to date with latest schema 2900.
10:46:14.956 PM	INFO	SolrSchemaUpgrade	
[main]: Current database upgrade ordinal is -1, latest upgrade step has ordinal 28
10:46:15.024 PM	INFO	AbstractSolrUpgrade	
[main]: Starting upgrade step: com.cloudera.nav.server.upgrade.DeleteEntities : Delete temporary entities.
10:46:15.031 PM	INFO	NavServerUtil	
[main]: Found 4 documents in solr core nav_elements
10:46:15.039 PM	INFO	NavServerUtil	
[main]: Found 1 documents in solr core nav_relations
10:46:15.042 PM	INFO	DeleteEntities	
[main]: Delete temporary relations query: {!join from=identity to=endpoint1Ids fromIndex=nav_elements}fileSystemPath:(*\/_SUCCESS OR *\/.trash OR \/user\/history OR *\/.staging OR \/tmp\/logs) AND sourceType:HDFS
10:46:15.206 PM	ERROR	SolrCore	
[qtp1100399456-26]: org.apache.solr.common.SolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker
	at org.apache.solr.search.CursorMark.<init>(CursorMark.java:104)
	at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:179)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at com.cloudera.nav.auth.SharedSecretRequestFilter.doFilter(SharedSecretRequestFilter.java:55)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)

10:46:15.221 PM	ERROR	NavServer	
[main]: Error upgrading solr configuration or data.
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
	at com.cloudera.nav.utils.solr.SolrResultSetIterator.getNextBatch(SolrResultSetIterator.java:96)
	at com.cloudera.nav.utils.solr.SolrResultSetIterator.hasNext(SolrResultSetIterator.java:76)
	at com.cloudera.nav.server.upgrade.DeleteEntities.deleteForSide(DeleteEntities.java:248)
	at com.cloudera.nav.server.upgrade.DeleteEntities.blindDelete(DeleteEntities.java:223)
	at com.cloudera.nav.server.upgrade.DeleteEntities.upgrade(DeleteEntities.java:76)
	at com.cloudera.nav.server.upgrade.AbstractSolrUpgrade.upgrade(AbstractSolrUpgrade.java:156)
	at com.cloudera.nav.server.SolrSchemaUpgrade.performJavaUpgrade(SolrSchemaUpgrade.java:400)
	at com.cloudera.nav.server.SolrSchemaUpgrade.upgradeSolr(SolrSchemaUpgrade.java:292)
	at com.cloudera.nav.server.NavServer.run(NavServer.java:434)
	at com.cloudera.nav.server.NavServer.main(NavServer.java:123)
10:46:15.329 PM	INFO	NavigatorInfrastructureEventDispatcher	
[pool-12-thread-1]: Waiting for Solr to Start.
10:46:15.331 PM	INFO	NavigatorInfrastructureEventDispatcher	
[pool-12-thread-1]: Succesfully started Solr.
10:46:15.332 PM	INFO	NavigatorInfrastructureEventDispatcher	
[pool-12-thread-1]: Waiting for Solr upgrade to complete.
10:46:15.489 PM	ERROR	NavServerUtil	
[pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details.
10:46:15.496 PM	INFO	NavServerUtil	
[pool-12-thread-1]: Exception occurred when checking for upgrade status.
java.lang.RuntimeException: Error encountered in fetching the udate progress information.
	at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88)
	at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281)
	at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231)
	at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158)
	at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:23)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
10:46:15.506 PM	INFO	SolrMaintenanceAware	
[pool-12-thread-1]: Checking for upgrade done status to be set.
10:46:15.514 PM	ERROR	NavServerUtil	
[pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details.
10:46:15.514 PM	INFO	NavServerUtil	
[pool-12-thread-1]: Exception occurred when checking for upgrade status.
java.lang.RuntimeException: Error encountered in fetching the udate progress information.
	at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88)
	at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281)
	at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231)
	at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158)
	at com.cloudera.nav.server.SolrMaintenanceAware.pollForUpgradeDone(SolrMaintenanceAware.java:34)
	at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:24)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
10:46:15.516 PM	INFO	SolrMaintenanceAware	
[pool-12-thread-1]: Upgrade status not set or available, continue checking for upgrade status to be set.

My backend is a PostgreSQL database.  Navigator Metastore Server keeps exiting and restarting as a result:

 

Unexpected Exits                                     Suppress...
This role encountered 4 unexpected exit(s) in the previous 5 minute(s). Critical threshold: any.
 
How could I solve this?  The other posts and solutions I could find were specific to MySQL only.  
 
Also only the servers are reporting and the Cloudera Management Service is installed.  Nothing else is installed on this Cluster yet.  

 

Thx,
TK

Champion
Posts: 777
Registered: ‎05-16-2016

Re: CM / CDH 6.2.X w/ PSQL:Cursor functionality requires a sort containing a uniqueKey field tie bre

Hi 

Take a peek in to the below , please try in your lower enviroment and validate before moving to production box. 

 

https://community.cloudera.com/t5/Data-Discovery-Optimization/Navigator-metadata-server-keep-on-cras...

Explorer
Posts: 18
Registered: ‎01-27-2019

Re: CM / CDH 6.2.X w/ PSQL:Cursor functionality requires a sort containing a uniqueKey field tie bre

Thank you but I saw that already before posting and tried the solutions from it.  However, I don't have these tables in PostgreSQL on CDH / CM 6.2.X:

 

"DELETE FROM NAV_UPGRADE_ORDINAL;" "INSERT INTO NAV_UPGRADE_ORDINAL VALUES (-1, -1);"

 

select * from NAV_UPGRADE_ORDINAL;

 

So I posted since on first impressions, this appears a unique case to 6.2.X and PostgreSQL.

 

postgres=# \c nav_mws01
Password for user postgres:
You are now connected to database "nav_mws01" as user "postgres".
nav_mws01=# \dt
List of relations
Schema | Name | Type | Owner
--------+--------------------------------+-------+-----------
public | activity_audit | table | nav_mws01
public | activity_last_poll_time | table | nav_mws01
public | audit_schema_version | table | nav_mws01
public | hbase_audit_events | table | nav_mws01
public | hbase_audit_events_2019_04_14 | table | nav_mws01
public | hbase_audit_events_2019_04_15 | table | nav_mws01
public | hbase_audit_events_2019_04_26 | table | nav_mws01
public | hbase_audit_events_2019_04_27 | table | nav_mws01
public | hbase_audit_events_2019_04_28 | table | nav_mws01
public | hdfs_audit_events | table | nav_mws01
public | hdfs_audit_events_2019_04_14 | table | nav_mws01
public | hdfs_audit_events_2019_04_15 | table | nav_mws01
public | hdfs_audit_events_2019_04_26 | table | nav_mws01
public | hdfs_audit_events_2019_04_27 | table | nav_mws01
public | hdfs_audit_events_2019_04_28 | table | nav_mws01
public | hive_audit_events | table | nav_mws01
public | hive_audit_events_2019_04_14 | table | nav_mws01
public | hive_audit_events_2019_04_15 | table | nav_mws01
public | hive_audit_events_2019_04_26 | table | nav_mws01
public | hive_audit_events_2019_04_27 | table | nav_mws01
public | hive_audit_events_2019_04_28 | table | nav_mws01
public | hue_audit_events | table | nav_mws01
public | hue_audit_events_2019_04_14 | table | nav_mws01
public | hue_audit_events_2019_04_15 | table | nav_mws01
public | hue_audit_events_2019_04_26 | table | nav_mws01
public | hue_audit_events_2019_04_27 | table | nav_mws01
public | hue_audit_events_2019_04_28 | table | nav_mws01
public | impala_audit_events | table | nav_mws01
public | impala_audit_events_2019_04_14 | table | nav_mws01
public | impala_audit_events_2019_04_15 | table | nav_mws01
public | impala_audit_events_2019_04_26 | table | nav_mws01
public | impala_audit_events_2019_04_27 | table | nav_mws01
public | impala_audit_events_2019_04_28 | table | nav_mws01
public | nav_count_staging_raw | table | nav_mws01
public | nav_file_info | table | nav_mws01
public | nav_file_info_staging | table | nav_mws01
public | nav_file_reads | table | nav_mws01
public | nav_file_reads_staging | table | nav_mws01
public | nav_file_reads_summary | table | nav_mws01
public | nav_parent_info | table | nav_mws01
public | nav_parent_info_staging | table | nav_mws01
public | nav_service_info | table | nav_mws01
public | nav_service_info_staging | table | nav_mws01
public | navms_audit_events | table | nav_mws01
public | navms_audit_events_2019_04_14 | table | nav_mws01
public | navms_audit_events_2019_04_15 | table | nav_mws01
public | navms_audit_events_2019_04_26 | table | nav_mws01
public | navms_audit_events_2019_04_27 | table | nav_mws01
public | navms_audit_events_2019_04_28 | table | nav_mws01
public | partition_info | table | nav_mws01
public | sentry_audit_events | table | nav_mws01
public | sentry_audit_events_2019_04_14 | table | nav_mws01
public | sentry_audit_events_2019_04_15 | table | nav_mws01
public | sentry_audit_events_2019_04_26 | table | nav_mws01
public | sentry_audit_events_2019_04_27 | table | nav_mws01
public | sentry_audit_events_2019_04_28 | table | nav_mws01
public | solr_audit_events | table | nav_mws01
public | solr_audit_events_2019_04_14 | table | nav_mws01
public | solr_audit_events_2019_04_15 | table | nav_mws01
public | solr_audit_events_2019_04_26 | table | nav_mws01
public | solr_audit_events_2019_04_27 | table | nav_mws01
public | solr_audit_events_2019_04_28 | table | nav_mws01
(62 rows)

nav_mws01=# select * from NAV_UPGRADE_ORDINAL;
ERROR: relation "nav_upgrade_ordinal" does not exist
LINE 1: select * from NAV_UPGRADE_ORDINAL;
^
nav_mws01=#

 

Thx,
TK