question Re: Metadata Server Web UI in Atlas responds with 500 Communication refused in Support Questions

Metadata Server Web UI in Atlas responds with 500 Communication refused

MarceloSaiedOK — Thu, 02 Nov 2017 04:05:12 GMT

HI,
I have upgraded from Ambari and HDP 2.5 to 2.6.3.
After installing Atlas, i get a warning , on the webUI , and the Web UI is not accesible.

at /var/log/atlas/application.log i can see that each time i request the url for the Atlas Dashboard at

http://xxxxxxxxxxx.com:21000/login.jsp i get an error 500

Thanks in advance

2017-11-01 16:23:08,686 WARN  - [pool-2-thread-7:] ~ /login.jsp (AbstractHttpConnection:552)
java.lang.NoClassDefFoundError: Could not initialize class org.apache.jasper.compiler.ErrorDispatcher
        at org.apache.jasper.compiler.Compiler.<init>(Compiler.java:139)
        at org.apache.jasper.JspCompilationContext.createCompiler(JspCompilationContext.java:288)
        at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:622)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:276)
        at org.eclipse.jetty.server.Dispatcher.error(Dispatcher.java:112)
        at org.eclipse.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:86)
        at org.eclipse.jetty.server.Response.sendError(Response.java:349)
        at org.eclipse.jetty.server.Response.sendError(Response.java:430)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:598)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

application.properties file

# Generated by Apache Ambari. Tue Oct 31 19:16:18 2017

atlas.audit.hbase.tablename=ATLAS_ENTITY_AUDIT_EVENTS
atlas.audit.hbase.zookeeper.quorum=xxxxxxxxxxxxxxxxxxxxxxxxx.com,xxxxxxxxxxxxxxxxxxxxxxxxx.com,xxxxxxxxxxxxxxxxxxxxxxxxx.com
atlas.audit.zookeeper.session.timeout.ms=60000
atlas.auth.policy.file=/usr/hdp/current/atlas-server/conf/policy-store.txt
atlas.authentication.keytab=/etc/security/keytabs/atlas.service.keytab
atlas.authentication.method.file=true
atlas.authentication.method.file.filename=/usr/hdp/current/atlas-server/conf/users-credentials.properties
atlas.authentication.method.kerberos=false
atlas.authentication.method.ldap=false
atlas.authentication.method.ldap.ad.base.dn=
atlas.authentication.method.ldap.ad.bind.dn=
atlas.authentication.method.ldap.ad.bind.password=
atlas.authentication.method.ldap.ad.default.role=ROLE_USER
atlas.authentication.method.ldap.ad.domain=
atlas.authentication.method.ldap.ad.referral=ignore
atlas.authentication.method.ldap.ad.url=
atlas.authentication.method.ldap.ad.user.searchfilter=(sAMAccountName={0})
atlas.authentication.method.ldap.base.dn=
atlas.authentication.method.ldap.bind.dn=
atlas.authentication.method.ldap.bind.password=
atlas.authentication.method.ldap.default.role=ROLE_USER
atlas.authentication.method.ldap.groupRoleAttribute=cn
atlas.authentication.method.ldap.groupSearchBase=
atlas.authentication.method.ldap.groupSearchFilter=
atlas.authentication.method.ldap.referral=ignore
atlas.authentication.method.ldap.type=ldap
atlas.authentication.method.ldap.url=
atlas.authentication.method.ldap.user.searchfilter=
atlas.authentication.method.ldap.userDNpattern=uid=
atlas.authentication.principal=atlas
atlas.authorizer.impl=simple
atlas.cluster.name=HADOOP_DESARROLLO
atlas.enableTLS=false
atlas.graph.index.search.backend=solr5
atlas.graph.index.search.solr.mode=cloud
atlas.graph.index.search.solr.zookeeper-url=xxxxxxxxxxxxxxx.com:2181/infra-solr,xxxxxxxxxxxxxxx.com:2181/infra-solr,xxxxxxxxxxxxxxxxxxxxxxx.com:2181/infra-solr
atlas.graph.storage.backend=hbase
atlas.graph.storage.hbase.table=atlas_titan
atlas.graph.storage.hostname=xxxxxxxxxxxxxxxxxxxxxxxxx.com,xxxxxxxxxxxxxxxxxxxxxxxxx.com,xxxxxxxxxxxxxxxxxxxxxxxxx.com
atlas.kafka.bootstrap.servers=xxxxxxxxxxxxxx.com:6667,xxxxxxxxxxxxxx.com:6667,xxxxxxxxxxxxxx.com:6667,xxxxxxxxxxxxxxxxxxxxxxxxx.com:6667
atlas.kafka.enable.auto.commit=false
atlas.kafka.hook.group.id=atlas
atlas.kafka.session.timeout.ms=30000
atlas.kafka.zookeeper.connect=xxxxxxxxxxxxxxxxxxxxxxxxx.com:2181,xxxxxxxxxxxxxxxxxxxxxxxxx.com:2181,xxxxxxxxxxxxxxxxxxxxxxxxx.com:2181
atlas.kafka.zookeeper.connection.timeout.ms=30000
atlas.kafka.zookeeper.session.timeout.ms=60000
atlas.kafka.zookeeper.sync.time.ms=20
atlas.lineage.schema.query.hive_table=hive_table where __guid='%s'\, columns
atlas.lineage.schema.query.Table=Table where __guid='%s'\, columns
atlas.notification.create.topics=true
atlas.notification.embedded=false
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.proxyusers=knox
atlas.rest.address=http://xxxxxxxxxxxxxxxxxxxxxxxxx.com:21000
atlas.server.address.id1=xxxxxxxxxxxxxxxxxxxxxxxxx.com:21000
atlas.server.bind.address=xxxxxxxxxxxxxxxxxxxxxxxxx.com
atlas.server.ha.enabled=false
atlas.server.http.port=21000
atlas.server.https.port=21443
atlas.server.ids=id1
atlas.solr.kerberos.enable=false
atlas.ssl.exclude.protocols=TLSv1.2
atlas.sso.knox.browser.useragent=
atlas.sso.knox.enabled=false
atlas.sso.knox.providerurl=https://xxxxxxxxxxxxxxxxxxxxxxxxx.com:8443/gateway/knoxsso/api/v1/websso
atlas.sso.knox.publicKey=

Alert :<br>HTTP 502 response from http://localhost:21000/api/atlas/admin/status in 0.003s (HTTP Error 502: Proxy Error ( Connection refused ))<img alt="">

Re: Metadata Server Web UI in Atlas responds with 500 Communication refused

amestry — Thu, 02 Nov 2017 07:52:38 GMT

Can you tell if you see this line in the logs?

jetty-8.1.19.v20160209 (Server:272)

From the log it appears that the classpath of the old version and the new version may be getting mixed up. This is causing Jetty not to serve up the pages correctly.

Is it possible to try this:

Stop Atlas via Ambari.
Ensure that /usr/hdp/current/atlas-server/server/webapp/atlas.war exists.
Remove this directory /usr/hdp/current/atlas-server/server/webapp/atlas
Start Atlas via Ambar

Re: Metadata Server Web UI in Atlas responds with 500 Communication refused

nahuel-tarello — Sat, 04 Nov 2017 02:37:54 GMT

Marcelo,

I just resolved the same issue, checking the error logs (atlas.yyyymmdd-HHMISS.err) I found this:

Exception in thread "main" org.apache.atlas.exception.AtlasBaseException: EmbeddedServer.Start: failed!
        at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:95)
        at org.apache.atlas.Atlas.main(Atlas.java:118)
Caused by: java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
        at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.eclipse.jetty.server.Server.doStart(Server.java:293)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
        at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:92)
        ... 1 more
log4j:WARN No appenders could be found for logger (org.eclipse.jetty.servlet.listener.ELContextCleaner).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

When I saw the message "Caused by: java.net.BindException: Address already in use" I realized that I had more than one instance of the Metadata Server running on host:

# ps -fea|grep atlas
atlas    11443     1  0 Oct31 ?        00:29:58 /usr/jdk64/jdk1.8.0_77/bin/java -Datlas.log.dir=/usr/hdp/2.6.3.0-235/atlas/logs -Datlas.log.file=application.log -Datlas.home=/usr/hdp/2.6.3.0-235/atlas -Datlas.conf=/usr/hdp/2.6.3.0-235/atlas/conf -Xms2048m -Xmx2048m -XX:MaxNewSize=600m -XX:MetaspaceSize=100m -XX:MaxMetaspaceSize=512m -server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/atlas_server.hprof -Xloggc:/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Dlog4j.configuration=atlas-log4j.xml -Djava.net.preferIPv4Stack=true -server -classpath /usr/hdp/2.6.3.0-235/atlas/conf:/usr/hdp/2.6.3.0-235/atlas/server/webapp/atlas/WEB-INF/classes:/usr/hdp/2.6.3.0-235/atlas/server/webapp/atlas/WEB-INF/lib/*:/usr/hdp/2.6.3.0-235/atlas/libext/*:/etc/hbase/conf org.apache.atlas.Atlas -app /usr/hdp/2.6.3.0-235/atlas/server/webapp/atlas
atlas    22076     1  0 Nov01 ?        00:24:03 /usr/jdk64/jdk1.8.0_77/bin/java -Datlas.log.dir=/var/log/atlas -Datlas.log.file=application.log -Datlas.home=/usr/hdp/2.6.3.0-235/atlas -Datlas.conf=/usr/hdp/current/atlas-server/conf -Xms2048m -Xmx2048m -XX:MaxNewSize=600m -XX:MetaspaceSize=100m -XX:MaxMetaspaceSize=512m -server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/atlas/atlas_server.hprof -Xloggc:/var/log/atlas/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Dlog4j.configuration=atlas-log4j.xml -classpath /usr/hdp/current/atlas-server/conf:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/classes:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib/*:/usr/hdp/2.6.3.0-235/atlas/libext/*:/etc/hbase/conf org.apache.atlas.Atlas -app /usr/hdp/current/atlas-server/server/webapp/atlas
atlas    23317     1 15 16:01 ?        00:01:05 /usr/jdk64/jdk1.8.0_77/bin/java -Datlas.log.dir=/var/log/atlas -Datlas.log.file=application.log -Datlas.home=/usr/hdp/2.6.3.0-235/atlas -Datlas.conf=/usr/hdp/current/atlas-server/conf -Xms2048m -Xmx2048m -XX:MaxNewSize=600m -XX:MetaspaceSize=100m -XX:MaxMetaspaceSize=512m -server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/atlas/atlas_server.hprof -Xloggc:/var/log/atlas/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -Dlog4j.configuration=atlas-log4j.xml -classpath /usr/hdp/current/atlas-server/conf:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/classes:/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib/*:/usr/hdp/2.6.3.0-235/atlas/libext/*:/etc/hbase/conf org.apache.atlas.Atlas -app /usr/hdp/current/atlas-server/server/webapp/atlas

Then I stopped the atlas services from ambari and killed all related processes on the host.

Now, the Atlas Web UI works without problems.

Let me know if you had the same results.

Re: Metadata Server Web UI in Atlas responds with 500 Communication refused

MarceloSaiedOK — Tue, 07 Nov 2017 23:20:23 GMT

Thanks for the lead 🙂