Cloudera Navigatror status 503

[#1]

Hi All.

We have an issue with cloudera navigator UI on  6.2 manual installation ..
Background..

Installed the navigator audit and metadat server roles on a cluster backed by external postgresql database

CDH is using auto_TLS

Navigator metadata and audit roles are status green on the cloudera manager UI

However when i browse to the navigator UI we get a status 503 service unavailable message.

we have confirmed the navigator role is up and running 

[root@cmhdgateway01 hue]# lsof -i:7187
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 12766 cloudera-scm 532u IPv4 814278 0t0 TCP *:7187 (LISTEN)
python2.7 17534 hue 31u IPv4 955050 0t0 TCP cmhdgateway01.cso.ie:50144->cmhdgateway01.cso.ie:7187 (CLOSE_WAIT)
[root@cmhdgateway01 hue]#

The following is captured on the startuplogs of navigator

2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vm.specification.version=1.8
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.arch.data.model=64
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.home=/usr/java/jdk1.8.0_181-cloudera/jre
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.java.command=com.cloudera.nav.server.NavServer /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cloudera-navigator.properties /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cloudera-navigator-cm-auth.properties /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/db.navms.properties /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cm-ext-accounts.properties
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.specification.vendor=Oracle Corporation
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: user.language=en
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: awt.toolkit=sun.awt.X11.XToolkit
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vm.info=mixed mode
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.version=1.8.0_181
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.ext.dirs=/usr/java/jdk1.8.0_181-cloudera/jre/lib/ext:/usr/java/packages/lib/ext
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.boot.class.path=/usr/java/jdk1.8.0_181-cloudera/jre/lib/resources.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/rt.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/jsse.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/jce.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/charsets.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/jfr.jar:/usr/java/jdk1.8.0_181-cloudera/jre/classes
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vendor=Oracle Corporation
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.awt.headless=true
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: file.separator=/
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vendor.url.bug=http://bugreport.sun.com/bugreport/
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: mgmt.log.file=mgmt-cmf-mgmt-NAVIGATORMETASERVER-cmhdgateway01.cso.ie.log.out
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.cpu.endian=little
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.io.unicode.encoding=UnicodeLittle
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: javax.net.ssl.trustStorePassword=PJx5BllHfNHUsaSGN6TP0uKDUqVkM9B5IqwgdQ8MlLA
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.cpu.isalist=
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: Maximum Memory allocated for JVM: 1.9675293 GB
2019-05-09 12:14:42,971 INFO com.cloudera.nav.server.NavServer [main]: Skipping the database initialization routine com.cloudera.nav.actions.MySQLEngineTypeVerifier as it is disabled.
2019-05-09 12:14:42,971 INFO com.cloudera.nav.server.NavServer [main]: Executing the database initialization routine com.cloudera.nav.actions.CharacterSetEncodingVerifier
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.CharacterSetEncodingVerifier [main]: JDBC Driver class org.postgresql.Driver
2019-05-09 12:14:43,196 INFO com.cloudera.nav.actions.CharacterSetEncodingVerifier [main]: Character set is: UTF8
2019-05-09 12:14:43,197 INFO com.cloudera.nav.server.NavServer [main]: Executing the database initialization routine com.cloudera.nav.server.UpgradeSchema
2019-05-09 12:14:43,297 INFO com.cloudera.enterprise.dbutil.DbUtil [main]: Schema version table already exists.
2019-05-09 12:14:43,300 INFO com.cloudera.enterprise.dbutil.DbUtil [main]: DB Schema version 60000.
2019-05-09 12:14:43,300 INFO com.cloudera.enterprise.dbutil.DbUtil [main]: Current database schema version: 60000
2019-05-09 12:14:43,468 INFO com.cloudera.nav.server.NavServer [main]: Skipping the database initialization routine com.cloudera.nav.server.UpgradeOrdinalVerifier as it is disabled.
2019-05-09 12:14:43,469 INFO com.cloudera.nav.server.NavServer [main]: Completed all database initialization routines successfully.
2019-05-09 12:14:43,792 INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource [main]: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> u0778a217d9g7lynla0g|15043a2f, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> org.postgresql.Driver, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, forceSynchronousCheckins -> false, forceUseNamedDriverClass -> false, identityToken -> u0778a217d9g7lynla0g|15043a2f, idleConnectionTestPeriod -> 300, initialPoolSize -> 2, jdbcUrl -> jdbc:postgresql://cmhdgateway01.cso.ie/navms, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, numHelperThreads -> 3, preferredTestQuery -> null, privilegeSpawnedThreads -> false, properties -> {user=******, password=******}, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ]
2019-05-09 12:14:43,826 WARN com.mchange.v2.resourcepool.BasicResourcePool [main]: Bad pool size config, start 2 < min 5. Using 5 as start.
2019-05-09 12:14:43,914 INFO com.cloudera.nav.server.NavServer [main]: Enabling SSL
2019-05-09 12:17:00,867 WARN org.eclipse.jetty.webapp.WebAppContext [main]: Failed startup of context o.e.j.w.WebAppContext@180da663{/,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-6.2.0.war-_-any-3567577415138419929.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-6.2.0.war-_-any-3567577415138419929.dir/webapp/],UNAVAILABLE}{/opt/cloudera/cm/cloudera-navigator-server/wars/nav-core-webapp-6.2.0.war}
java.net.ConnectException: Connection timed out (Connection timed out)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) 
       at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
        at sun.net.www.http.HttpClient.New(HttpClient.java:339)
        at sun.net.www.http.HttpClient.New(HttpClient.java:357)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:647)
        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1304)
        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1270)
        at com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:264)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1161)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
        at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
        at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
        at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:327)
        at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:255)
        at org.eclipse.jetty.webapp.Descriptor.parse(Descriptor.java:55)
        at org.eclipse.jetty.webapp.WebDescriptor.parse(WebDescriptor.java:212)
        at org.eclipse.jetty.webapp.MetaData.setWebXml(MetaData.java:194)
        at org.eclipse.jetty.webapp.WebXmlConfiguration.preConfigure(WebXmlConfiguration.java:60)
        at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:506)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:544)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:168)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
        at org.eclipse.jetty.server.Server.start(Server.java:415)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
        at org.eclipse.jetty.server.Server.doStart(Server.java:382)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at com.cloudera.nav.server.NavServer.run(NavServer.java:425)
        at com.cloudera.nav.server.NavServer.main(NavServer.java:123)
2019-05-09 12:17:04,387 WARN org.eclipse.jetty.security.SecurityHandler [main]: ServletContext@o.e.j.w.WebAppContext@710b18a6{/solr,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-1169072435507539588.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-1169072435507539588.dir/webapp/],STARTING}{/opt/cloudera/cm/cloudera-navigator-server/wars/solr-4.10.3.war} has uncovered http methods for path: /*
2019-05-09 12:17:07,333 WARN org.apache.solr.schema.IndexSchema [coreLoadExecutor-5-thread-1]: Field user is not multivalued and destination for multiple copyFields (2)
2019-05-09 12:17:07,969 WARN org.eclipse.jetty.util.ssl.SslContextFactory.config [main]: No Client EndPointIdentificationAlgorithm configured for SslContextFactory@585ac855[provider=null,keyStore=file:///run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cm-auto-host_keystore.jks,trustStore=null]
2019-05-09 12:17:10,241 INFO com.cloudera.nav.server.NavServerUtil [main]: No nav_elements entities exist with id field in solr
2019-05-09 12:17:10,279 INFO com.cloudera.nav.server.NavServerUtil [main]: No nav_relations entities exist with id field in solr
2019-05-09 12:17:10,525 INFO com.cloudera.nav.server.NavServerUtil [main]: Found 0 documents in solr core nav_elements
2019-05-09 12:17:10,534 INFO com.cloudera.nav.server.NavServerUtil [main]: Found 0 documents in solr core nav_relations
2019-05-09 12:17:10,534 INFO com.cloudera.nav.server.SolrSchemaUpgrade [main]: Checking if solr schema upgrade is needed.
2019-05-09 12:17:10,550 INFO com.cloudera.nav.server.SolrSchemaUpgrade [main]: Schema version 2900 already up to date with latest schema 2900.
2019-05-09 12:17:10,559 INFO com.cloudera.nav.server.SolrSchemaUpgrade [main]: Current database upgrade ordinal is 28, latest upgrade step has ordinal 28
2019-05-09 12:17:10,733 INFO com.cloudera.nav.server.NavServer [main]: Cleaning up maintenance history of any previously running jobs.
2019-05-09 12:17:10,752 INFO com.cloudera.nav.server.NavServer [main]: Navigator Metadata Server listening on https://3.1.2.116:7187

[#2 · Accepted solution]

Hi @Ryanp ,

Does your cluster happen to be isolated from Internet? If so, then you may hit a known issue:

In Cloudera Manager 6.2.0, if the Navigator Metadata server host doesn't have access to the internet, the Metadata data server role will fail to start.

We have a knowledge article on this topic and it outlines the workaround steps to fix the issue:

https://community.cloudera.com/t5/Customer/Navigator-Metadata-Server-Doesn-t-start-on-isolated-clust...

Thanks,

Li

[#3 · Accepted solution]

Hi @Ryanp ,

Does your cluster happen to be isolated from Internet? If so, then you may hit a known issue:

In Cloudera Manager 6.2.0, if the Navigator Metadata server host doesn't have access to the internet, the Metadata data server role will fail to start.

We have a knowledge article on this topic and it outlines the workaround steps to fix the issue:

https://community.cloudera.com/t5/Customer/Navigator-Metadata-Server-Doesn-t-start-on-isolated-clust...

Thanks,

Li

[#4]

Hi li
You are correct .. the machine is isolated from internet..
Appreciate the response and help..
Unfortunately I can not access the knowledge article..
Is there a public resource I can read it from
Regards
P

[#5]

Hi li
You are correct .. the machine is isolated from internet..
Appreciate the response and help..
Unfortunately I can not access the knowledge article..
Is there a public resource I can read it from
Regards
P

[#6]

Hi @Ryanp ,

Here are the snippet from the knowledge article:

A workaround can be achieved using the steps below. (NMS=Navigator Metadata Server)

1. Identify IP address of the NMS. (can be gathered in Cloudera Management services->instances->Navigator Metadata Server)

2. Add the following line to the bottom of /etc/hosts on the NMS

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
<NMS_IP_ADDRESS> java.sun.com

3. On NMS, install httpd, depending on your Linux distribution. As this host does not have internet access, you will need to manually import and install rpms.

https://httpd.apache.org/docs/2.4/platform/rpm.html

4. Make a directory for java dtd file, and adjust permissions.

mkdir /var/www/html/dtd; chown 755 /var/www/html/dtd

5. Acquire Java DTD file from the following URL.

http://java.sun.com/dtd/web-app_2_3.dtd

6. Copy the file to /var/www/html/dtd directory on the NMS.  Ensure the file is readable by "other".

7. Enable httpd server

sudo systemctl enable httpd; sudo systemctl start httpd

8. Start NMS.

NOTE: httpd must run for each NMS restart, this is not a one-off workaround. Systemctl command should ensure that this is the case.   If the above workaround is not possible, a patch is also available from Cloudera Support until this is resolved in an upcoming release of Cloudera Manager.

Thanks and hope this helps,

Li

[#7]

HI Li.

Appreciate the response . 

I whitelisted the java.sun.com  and oracle.com fqdn here and have  navigator coming online

thank you very much

[#8 · Accepted solution]

Hi @Ryanp ,

You are very welcome and glad to hear the issue is resolved!

Thanks,

Li