Support Questions
Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

CM / CDH 6.2.X w/ PSQL:Cursor functionality requires a sort containing a uniqueKey field tie breaker

TCloud
Explorer

Created on ‎04-27-2019 08:48 PM - last edited on ‎04-28-2019 05:26 AM by Community Manager Community Manager

Hey All,

 

I'm installing a vanilla CM / CDH 6.2.X and running into the issue below from the Navigator Metadata Server:

 

10:45:54.757 PM	INFO	JsonModelLoader	
[main]: Added audit event model for SENTRY
10:45:54.761 PM	INFO	JsonModelLoader	
[main]: Added audit event model for SOLR
10:46:10.631 PM	WARN	SecurityHandler	
[main]: ServletContext@o.e.j.w.WebAppContext@241e8ea6{/solr,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webapp/],STARTING}{/opt/cloudera/cm/cloudera-navigator-server/wars/solr-4.10.3.war} has uncovered http methods for path: /*
10:46:12.171 PM	WARN	IndexSchema	
[coreLoadExecutor-5-thread-1]: Field user is not multivalued and destination for multiple copyFields (2)
10:46:14.883 PM	INFO	NavServerUtil	
[main]: Found 4 documents in solr core nav_elements
10:46:14.895 PM	INFO	NavServerUtil	
[main]: Found 1 documents in solr core nav_relations
10:46:14.896 PM	INFO	SolrSchemaUpgrade	
[main]: Checking if solr schema upgrade is needed.
10:46:14.928 PM	INFO	SolrSchemaUpgrade	
[main]: Schema version 2900 already up to date with latest schema 2900.
10:46:14.956 PM	INFO	SolrSchemaUpgrade	
[main]: Current database upgrade ordinal is -1, latest upgrade step has ordinal 28
10:46:15.024 PM	INFO	AbstractSolrUpgrade	
[main]: Starting upgrade step: com.cloudera.nav.server.upgrade.DeleteEntities : Delete temporary entities.
10:46:15.031 PM	INFO	NavServerUtil	
[main]: Found 4 documents in solr core nav_elements
10:46:15.039 PM	INFO	NavServerUtil	
[main]: Found 1 documents in solr core nav_relations
10:46:15.042 PM	INFO	DeleteEntities	
[main]: Delete temporary relations query: {!join from=identity to=endpoint1Ids fromIndex=nav_elements}fileSystemPath:(*\/_SUCCESS OR *\/.trash OR \/user\/history OR *\/.staging OR \/tmp\/logs) AND sourceType:HDFS
10:46:15.206 PM	ERROR	SolrCore	
[qtp1100399456-26]: org.apache.solr.common.SolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker
	at org.apache.solr.search.CursorMark.<init>(CursorMark.java:104)
	at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:179)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at com.cloudera.nav.auth.SharedSecretRequestFilter.doFilter(SharedSecretRequestFilter.java:55)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:502)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
	at java.lang.Thread.run(Thread.java:748)

10:46:15.221 PM	ERROR	NavServer	
[main]: Error upgrading solr configuration or data.
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310)
	at com.cloudera.nav.utils.solr.SolrResultSetIterator.getNextBatch(SolrResultSetIterator.java:96)
	at com.cloudera.nav.utils.solr.SolrResultSetIterator.hasNext(SolrResultSetIterator.java:76)
	at com.cloudera.nav.server.upgrade.DeleteEntities.deleteForSide(DeleteEntities.java:248)
	at com.cloudera.nav.server.upgrade.DeleteEntities.blindDelete(DeleteEntities.java:223)
	at com.cloudera.nav.server.upgrade.DeleteEntities.upgrade(DeleteEntities.java:76)
	at com.cloudera.nav.server.upgrade.AbstractSolrUpgrade.upgrade(AbstractSolrUpgrade.java:156)
	at com.cloudera.nav.server.SolrSchemaUpgrade.performJavaUpgrade(SolrSchemaUpgrade.java:400)
	at com.cloudera.nav.server.SolrSchemaUpgrade.upgradeSolr(SolrSchemaUpgrade.java:292)
	at com.cloudera.nav.server.NavServer.run(NavServer.java:434)
	at com.cloudera.nav.server.NavServer.main(NavServer.java:123)
10:46:15.329 PM	INFO	NavigatorInfrastructureEventDispatcher	
[pool-12-thread-1]: Waiting for Solr to Start.
10:46:15.331 PM	INFO	NavigatorInfrastructureEventDispatcher	
[pool-12-thread-1]: Succesfully started Solr.
10:46:15.332 PM	INFO	NavigatorInfrastructureEventDispatcher	
[pool-12-thread-1]: Waiting for Solr upgrade to complete.
10:46:15.489 PM	ERROR	NavServerUtil	
[pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details.
10:46:15.496 PM	INFO	NavServerUtil	
[pool-12-thread-1]: Exception occurred when checking for upgrade status.
java.lang.RuntimeException: Error encountered in fetching the udate progress information.
	at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88)
	at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281)
	at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231)
	at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158)
	at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:23)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
10:46:15.506 PM	INFO	SolrMaintenanceAware	
[pool-12-thread-1]: Checking for upgrade done status to be set.
10:46:15.514 PM	ERROR	NavServerUtil	
[pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details.
10:46:15.514 PM	INFO	NavServerUtil	
[pool-12-thread-1]: Exception occurred when checking for upgrade status.
java.lang.RuntimeException: Error encountered in fetching the udate progress information.
	at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88)
	at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281)
	at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231)
	at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158)
	at com.cloudera.nav.server.SolrMaintenanceAware.pollForUpgradeDone(SolrMaintenanceAware.java:34)
	at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:24)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35)
	at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
10:46:15.516 PM	INFO	SolrMaintenanceAware	
[pool-12-thread-1]: Upgrade status not set or available, continue checking for upgrade status to be set.

My backend is a PostgreSQL database.  Navigator Metastore Server keeps exiting and restarting as a result:

 

Unexpected Exits                                     Suppress...
This role encountered 4 unexpected exit(s) in the previous 5 minute(s). Critical threshold: any.
 
How could I solve this?  The other posts and solutions I could find were specific to MySQL only.  
 
Also only the servers are reporting and the Cloudera Management Service is installed.  Nothing else is installed on this Cluster yet.  

 

Thx,
TK

1,120 Views
0 Kudos
2 REPLIES 2

csguna
Champion

Created ‎04-28-2019 08:04 AM

Hi 

Take a peek in to the below , please try in your lower enviroment and validate before moving to production box. 

 

https://community.cloudera.com/t5/Data-Discovery-Optimization/Navigator-metadata-server-keep-on-cras...

1,109 Views
0 Kudos

TCloud
Explorer

Created ‎04-28-2019 09:22 AM

Thank you but I saw that already before posting and tried the solutions from it.  However, I don't have these tables in PostgreSQL on CDH / CM 6.2.X:

 

"DELETE FROM NAV_UPGRADE_ORDINAL;" "INSERT INTO NAV_UPGRADE_ORDINAL VALUES (-1, -1);"

 

select * from NAV_UPGRADE_ORDINAL;

 

So I posted since on first impressions, this appears a unique case to 6.2.X and PostgreSQL.

 

postgres=# \c nav_mws01
Password for user postgres:
You are now connected to database "nav_mws01" as user "postgres".
nav_mws01=# \dt
List of relations
Schema | Name | Type | Owner
--------+--------------------------------+-------+-----------
public | activity_audit | table | nav_mws01
public | activity_last_poll_time | table | nav_mws01
public | audit_schema_version | table | nav_mws01
public | hbase_audit_events | table | nav_mws01
public | hbase_audit_events_2019_04_14 | table | nav_mws01
public | hbase_audit_events_2019_04_15 | table | nav_mws01
public | hbase_audit_events_2019_04_26 | table | nav_mws01
public | hbase_audit_events_2019_04_27 | table | nav_mws01
public | hbase_audit_events_2019_04_28 | table | nav_mws01
public | hdfs_audit_events | table | nav_mws01
public | hdfs_audit_events_2019_04_14 | table | nav_mws01
public | hdfs_audit_events_2019_04_15 | table | nav_mws01
public | hdfs_audit_events_2019_04_26 | table | nav_mws01
public | hdfs_audit_events_2019_04_27 | table | nav_mws01
public | hdfs_audit_events_2019_04_28 | table | nav_mws01
public | hive_audit_events | table | nav_mws01
public | hive_audit_events_2019_04_14 | table | nav_mws01
public | hive_audit_events_2019_04_15 | table | nav_mws01
public | hive_audit_events_2019_04_26 | table | nav_mws01
public | hive_audit_events_2019_04_27 | table | nav_mws01
public | hive_audit_events_2019_04_28 | table | nav_mws01
public | hue_audit_events | table | nav_mws01
public | hue_audit_events_2019_04_14 | table | nav_mws01
public | hue_audit_events_2019_04_15 | table | nav_mws01
public | hue_audit_events_2019_04_26 | table | nav_mws01
public | hue_audit_events_2019_04_27 | table | nav_mws01
public | hue_audit_events_2019_04_28 | table | nav_mws01
public | impala_audit_events | table | nav_mws01
public | impala_audit_events_2019_04_14 | table | nav_mws01
public | impala_audit_events_2019_04_15 | table | nav_mws01
public | impala_audit_events_2019_04_26 | table | nav_mws01
public | impala_audit_events_2019_04_27 | table | nav_mws01
public | impala_audit_events_2019_04_28 | table | nav_mws01
public | nav_count_staging_raw | table | nav_mws01
public | nav_file_info | table | nav_mws01
public | nav_file_info_staging | table | nav_mws01
public | nav_file_reads | table | nav_mws01
public | nav_file_reads_staging | table | nav_mws01
public | nav_file_reads_summary | table | nav_mws01
public | nav_parent_info | table | nav_mws01
public | nav_parent_info_staging | table | nav_mws01
public | nav_service_info | table | nav_mws01
public | nav_service_info_staging | table | nav_mws01
public | navms_audit_events | table | nav_mws01
public | navms_audit_events_2019_04_14 | table | nav_mws01
public | navms_audit_events_2019_04_15 | table | nav_mws01
public | navms_audit_events_2019_04_26 | table | nav_mws01
public | navms_audit_events_2019_04_27 | table | nav_mws01
public | navms_audit_events_2019_04_28 | table | nav_mws01
public | partition_info | table | nav_mws01
public | sentry_audit_events | table | nav_mws01
public | sentry_audit_events_2019_04_14 | table | nav_mws01
public | sentry_audit_events_2019_04_15 | table | nav_mws01
public | sentry_audit_events_2019_04_26 | table | nav_mws01
public | sentry_audit_events_2019_04_27 | table | nav_mws01
public | sentry_audit_events_2019_04_28 | table | nav_mws01
public | solr_audit_events | table | nav_mws01
public | solr_audit_events_2019_04_14 | table | nav_mws01
public | solr_audit_events_2019_04_15 | table | nav_mws01
public | solr_audit_events_2019_04_26 | table | nav_mws01
public | solr_audit_events_2019_04_27 | table | nav_mws01
public | solr_audit_events_2019_04_28 | table | nav_mws01
(62 rows)

nav_mws01=# select * from NAV_UPGRADE_ORDINAL;
ERROR: relation "nav_upgrade_ordinal" does not exist
LINE 1: select * from NAV_UPGRADE_ORDINAL;
^
nav_mws01=#

 

Thx,
TK

1,101 Views
0 Kudos
Take a Tour of the Community
Don't have an account?
Register
Your experience may be limited. Sign in to explore more.
Announcements
Product Announcements
CDP Public Cloud Release Summary: June 2022
What's New @ Cloudera
Cloudera DataFlow for the Public Cloud 2.1 introduces in-place upgrades as a Technical Preview feature
What's New @ Cloudera
CDP Operational Database (COD) supports CDP Control Planes for multiple regions.
What's New @ Cloudera
CDP Operational Database (COD) supports Multiple Availability Zones (Multi-AZ) on AWS
What's New @ Cloudera
COD now supports the “Store File Tracking” (SFT) as a general availability feature.
View More Announcements
; ;