Skip to main content
Ryanp
Ryanp
Gold ⭐️
May 9, 2019
Solved

Cloudera Navigatror status 503

Hi All.

We have an issue with cloudera navigator UI on  6.2 manual installation ..
Background..

Installed the navigator audit and metadat server roles on a cluster backed by external postgresql database

CDH is using auto_TLS

Navigator metadata and audit roles are status green on the cloudera manager UI

 

However when i browse to the navigator UI we get a status 503 service unavailable message.

nav error.PNG

 

we have confirmed the navigator role is up and running 

[root@cmhdgateway01 hue]# lsof -i:7187
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 12766 cloudera-scm 532u IPv4 814278 0t0 TCP *:7187 (LISTEN)
python2.7 17534 hue 31u IPv4 955050 0t0 TCP cmhdgateway01.cso.ie:50144->cmhdgateway01.cso.ie:7187 (CLOSE_WAIT)
[root@cmhdgateway01 hue]#

 

The following is captured on the startuplogs of navigator

2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vm.specification.version=1.8
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.arch.data.model=64
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.home=/usr/java/jdk1.8.0_181-cloudera/jre
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.java.command=com.cloudera.nav.server.NavServer /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cloudera-navigator.properties /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cloudera-navigator-cm-auth.properties /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/db.navms.properties /run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cm-ext-accounts.properties
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.specification.vendor=Oracle Corporation
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: user.language=en
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: awt.toolkit=sun.awt.X11.XToolkit
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vm.info=mixed mode
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.version=1.8.0_181
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.ext.dirs=/usr/java/jdk1.8.0_181-cloudera/jre/lib/ext:/usr/java/packages/lib/ext
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.boot.class.path=/usr/java/jdk1.8.0_181-cloudera/jre/lib/resources.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/rt.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/sunrsasign.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/jsse.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/jce.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/charsets.jar:/usr/java/jdk1.8.0_181-cloudera/jre/lib/jfr.jar:/usr/java/jdk1.8.0_181-cloudera/jre/classes
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vendor=Oracle Corporation
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.awt.headless=true
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: file.separator=/
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: java.vendor.url.bug=http://bugreport.sun.com/bugreport/
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: mgmt.log.file=mgmt-cmf-mgmt-NAVIGATORMETASERVER-cmhdgateway01.cso.ie.log.out
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.cpu.endian=little
2019-05-09 12:14:42,970 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.io.unicode.encoding=UnicodeLittle
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: javax.net.ssl.trustStorePassword=PJx5BllHfNHUsaSGN6TP0uKDUqVkM9B5IqwgdQ8MlLA
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: sun.cpu.isalist=
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.LogEnvironmentValues [main]: Maximum Memory allocated for JVM: 1.9675293 GB
2019-05-09 12:14:42,971 INFO com.cloudera.nav.server.NavServer [main]: Skipping the database initialization routine com.cloudera.nav.actions.MySQLEngineTypeVerifier as it is disabled.
2019-05-09 12:14:42,971 INFO com.cloudera.nav.server.NavServer [main]: Executing the database initialization routine com.cloudera.nav.actions.CharacterSetEncodingVerifier
2019-05-09 12:14:42,971 INFO com.cloudera.nav.actions.CharacterSetEncodingVerifier [main]: JDBC Driver class org.postgresql.Driver
2019-05-09 12:14:43,196 INFO com.cloudera.nav.actions.CharacterSetEncodingVerifier [main]: Character set is: UTF8
2019-05-09 12:14:43,197 INFO com.cloudera.nav.server.NavServer [main]: Executing the database initialization routine com.cloudera.nav.server.UpgradeSchema
2019-05-09 12:14:43,297 INFO com.cloudera.enterprise.dbutil.DbUtil [main]: Schema version table already exists.
2019-05-09 12:14:43,300 INFO com.cloudera.enterprise.dbutil.DbUtil [main]: DB Schema version 60000.
2019-05-09 12:14:43,300 INFO com.cloudera.enterprise.dbutil.DbUtil [main]: Current database schema version: 60000
2019-05-09 12:14:43,468 INFO com.cloudera.nav.server.NavServer [main]: Skipping the database initialization routine com.cloudera.nav.server.UpgradeOrdinalVerifier as it is disabled.
2019-05-09 12:14:43,469 INFO com.cloudera.nav.server.NavServer [main]: Completed all database initialization routines successfully.
2019-05-09 12:14:43,792 INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource [main]: Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, contextClassLoaderSource -> caller, dataSourceName -> u0778a217d9g7lynla0g|15043a2f, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> org.postgresql.Driver, extensions -> {}, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, forceSynchronousCheckins -> false, forceUseNamedDriverClass -> false, identityToken -> u0778a217d9g7lynla0g|15043a2f, idleConnectionTestPeriod -> 300, initialPoolSize -> 2, jdbcUrl -> jdbc:postgresql://cmhdgateway01.cso.ie/navms, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, numHelperThreads -> 3, preferredTestQuery -> null, privilegeSpawnedThreads -> false, properties -> {user=******, password=******}, propertyCycle -> 0, statementCacheNumDeferredCloseThreads -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, userOverrides -> {}, usesTraditionalReflectiveProxies -> false ]
2019-05-09 12:14:43,826 WARN com.mchange.v2.resourcepool.BasicResourcePool [main]: Bad pool size config, start 2 < min 5. Using 5 as start.
2019-05-09 12:14:43,914 INFO com.cloudera.nav.server.NavServer [main]: Enabling SSL
2019-05-09 12:17:00,867 WARN org.eclipse.jetty.webapp.WebAppContext [main]: Failed startup of context o.e.j.w.WebAppContext@180da663{/,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-6.2.0.war-_-any-3567577415138419929.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-nav-core-webapp-6.2.0.war-_-any-3567577415138419929.dir/webapp/],UNAVAILABLE}{/opt/cloudera/cm/cloudera-navigator-server/wars/nav-core-webapp-6.2.0.war}
java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:647)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startEntity(XMLEntityManager.java:1304)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.startDTDEntity(XMLEntityManager.java:1270)
at com.sun.org.apache.xerces.internal.impl.XMLDTDScannerImpl.setInputSource(XMLDTDScannerImpl.java:264)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.dispatch(XMLDocumentScannerImpl.java:1161)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$DTDDriver.next(XMLDocumentScannerImpl.java:1045)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(XMLDocumentScannerImpl.java:959)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:327)
at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:255)
at org.eclipse.jetty.webapp.Descriptor.parse(Descriptor.java:55)
at org.eclipse.jetty.webapp.WebDescriptor.parse(WebDescriptor.java:212)
at org.eclipse.jetty.webapp.MetaData.setWebXml(MetaData.java:194)
at org.eclipse.jetty.webapp.WebXmlConfiguration.preConfigure(WebXmlConfiguration.java:60)
at org.eclipse.jetty.webapp.WebAppContext.preConfigure(WebAppContext.java:506)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:544)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:168)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138)
at org.eclipse.jetty.server.Server.start(Server.java:415)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:108)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
at org.eclipse.jetty.server.Server.doStart(Server.java:382)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at com.cloudera.nav.server.NavServer.run(NavServer.java:425)
at com.cloudera.nav.server.NavServer.main(NavServer.java:123)
2019-05-09 12:17:04,387 WARN org.eclipse.jetty.security.SecurityHandler [main]: ServletContext@o.e.j.w.WebAppContext@710b18a6{/solr,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-1169072435507539588.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-1169072435507539588.dir/webapp/],STARTING}{/opt/cloudera/cm/cloudera-navigator-server/wars/solr-4.10.3.war} has uncovered http methods for path: /*
2019-05-09 12:17:07,333 WARN org.apache.solr.schema.IndexSchema [coreLoadExecutor-5-thread-1]: Field user is not multivalued and destination for multiple copyFields (2)
2019-05-09 12:17:07,969 WARN org.eclipse.jetty.util.ssl.SslContextFactory.config [main]: No Client EndPointIdentificationAlgorithm configured for SslContextFactory@585ac855[provider=null,keyStore=file:///run/cloudera-scm-agent/process/420-cloudera-mgmt-NAVIGATORMETASERVER/cm-auto-host_keystore.jks,trustStore=null]
2019-05-09 12:17:10,241 INFO com.cloudera.nav.server.NavServerUtil [main]: No nav_elements entities exist with id field in solr
2019-05-09 12:17:10,279 INFO com.cloudera.nav.server.NavServerUtil [main]: No nav_relations entities exist with id field in solr
2019-05-09 12:17:10,525 INFO com.cloudera.nav.server.NavServerUtil [main]: Found 0 documents in solr core nav_elements
2019-05-09 12:17:10,534 INFO com.cloudera.nav.server.NavServerUtil [main]: Found 0 documents in solr core nav_relations
2019-05-09 12:17:10,534 INFO com.cloudera.nav.server.SolrSchemaUpgrade [main]: Checking if solr schema upgrade is needed.
2019-05-09 12:17:10,550 INFO com.cloudera.nav.server.SolrSchemaUpgrade [main]: Schema version 2900 already up to date with latest schema 2900.
2019-05-09 12:17:10,559 INFO com.cloudera.nav.server.SolrSchemaUpgrade [main]: Current database upgrade ordinal is 28, latest upgrade step has ordinal 28
2019-05-09 12:17:10,733 INFO com.cloudera.nav.server.NavServer [main]: Cleaning up maintenance history of any previously running jobs.
2019-05-09 12:17:10,752 INFO com.cloudera.nav.server.NavServer [main]: Navigator Metadata Server listening on https://3.1.2.116:7187

 

 

lwang
lwangAnswer
Platinum ⭐️
May 9, 2019

Hi @Ryanp ,

 

Does your cluster happen to be isolated from Internet? If so, then you may hit a known issue:

In Cloudera Manager 6.2.0, if the Navigator Metadata server host doesn't have access to the internet, the Metadata data server role will fail to start.

 

We have a knowledge article on this topic and it outlines the workaround steps to fix the issue:

https://community.cloudera.com/t5/Customer/Navigator-Metadata-Server-Doesn-t-start-on-isolated-cluster/ta-p/89479

 

Thanks,

Li

Li Wang, Technical Solution Manager
View in thread

6 replies

lwang
Platinum ⭐️
lwangAnswer
Platinum ⭐️
May 9, 2019

Hi @Ryanp ,

 

Does your cluster happen to be isolated from Internet? If so, then you may hit a known issue:

In Cloudera Manager 6.2.0, if the Navigator Metadata server host doesn't have access to the internet, the Metadata data server role will fail to start.

 

We have a knowledge article on this topic and it outlines the workaround steps to fix the issue:

https://community.cloudera.com/t5/Customer/Navigator-Metadata-Server-Doesn-t-start-on-isolated-cluster/ta-p/89479

 

Thanks,

Li

Translate
Li Wang, Technical Solution Manager
    Ryanp
    RyanpAuthor
    Gold ⭐️
    May 9, 2019
    Hi li
    You are correct .. the machine is isolated from internet..
    Appreciate the response and help..
    Unfortunately I can not access the knowledge article..
    Is there a public resource I can read it from
    Regards
    P
    Translate
    Ryanp
    RyanpAuthor
    Gold ⭐️
    May 9, 2019
    Hi li
    You are correct .. the machine is isolated from internet..
    Appreciate the response and help..
    Unfortunately I can not access the knowledge article..
    Is there a public resource I can read it from
    Regards
    P
    Translate
    lwang
    Platinum ⭐️
    lwang
    Platinum ⭐️
    May 9, 2019

    Hi @Ryanp ,

     

    Here are the snippet from the knowledge article:

     

    A workaround can be achieved using the steps below. (NMS=Navigator Metadata Server)

     

    1. Identify IP address of the NMS. (can be gathered in Cloudera Management services->instances->Navigator Metadata Server)

     

    2. Add the following line to the bottom of /etc/hosts on the NMS

    127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
    <NMS_IP_ADDRESS> java.sun.com

     

    3. On NMS, install httpd, depending on your Linux distribution. As this host does not have internet access, you will need to manually import and install rpms.

    https://httpd.apache.org/docs/2.4/platform/rpm.html

     

    4. Make a directory for java dtd file, and adjust permissions.

    mkdir /var/www/html/dtd; chown 755 /var/www/html/dtd

     

    5. Acquire Java DTD file from the following URL.

     

    http://java.sun.com/dtd/web-app_2_3.dtd

    6. Copy the file to /var/www/html/dtd directory on the NMS.  Ensure the file is readable by "other".

     

    7. Enable httpd server

    sudo systemctl enable httpd; sudo systemctl start httpd

     

    8. Start NMS.

     

    NOTE: httpd must run for each NMS restart, this is not a one-off workaround. Systemctl command should ensure that this is the case.   If the above workaround is not possible, a patch is also available from Cloudera Support until this is resolved in an upcoming release of Cloudera Manager.

     

    Thanks and hope this helps,

    Li

    Translate
    Li Wang, Technical Solution Manager
      Ryanp
      RyanpAuthor
      Gold ⭐️
      May 10, 2019

      HI Li.

      Appreciate the response . 

      I whitelisted the java.sun.com  and oracle.com fqdn here and have  navigator coming online

      thank you very much

      Translate
      Terms & ConditionsAccessibility statement