Atlas Metadata Server error HTTP 503 response from http://localhost:21000/api/atlas/admin/status in 0.000s (HTTP Error 503: Service Unavailable)

[#1]

When I upgraded hdp from 2.5.3 to 2.6.0.3 then I am getting following warning in Ambari:

HTTP 503 response from http://localhost:21000/api/atlas/admin/status in 0.000s (HTTP Error 503: Service Unavailable)

Also following error in application.log file

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo} span.s1 {font-variant-ligatures: no-common-ligatures} span.s2 {font-variant-ligatures: no-common-ligatures; background-color: #e6e600}

2017-05-12 00:06:43,877 INFO - [qtp2050835901-1065 - 0d5633f1-091f-47ac-949e-ed1f3583122e:] ~ Audit: UNKNOWN/127.0.0.1-127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:06Z (AUDIT:104)

2017-05-12 00:07:44,005 INFO - [qtp2050835901-1066 - 329b25a9-f9c9-4142-ae11-42f9faed4e7b:] ~ Audit: UNKNOWN/127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:07Z (AuditFilter:91)

2017-05-12 00:07:44,006 INFO - [qtp2050835901-1066 - 329b25a9-f9c9-4142-ae11-42f9faed4e7b:] ~ Audit: UNKNOWN/127.0.0.1-127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:07Z (AUDIT:104)

2017-05-12 00:08:43,872 INFO - [qtp2050835901-1066 - 2e95c961-12cc-4353-a360-32c1f40f6678:] ~ Audit: UNKNOWN/127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:08Z (AuditFilter:91)

[#2]

@Saurabh

Please check if your Atlas Service is running fine on port 21000 on localhost?

# netstat -tnlpa | grep  21000

.

Also do you see any ERROR/ Warning on atlas log?

# less /var/log/atlas/atlas.20170419-185147.err
# less /var/log/atlas/atlas.20170419-185147.out 

.

[#3]

I see following logs in those files.

netstat -tnlpa | grep 21000

(No info could be read for "-p": geteuid()=4569 but you should be root.)

tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN -

cat atlas.20170512-081144.out

{metadata.broker.list=<server>:6667, request.timeout.ms=30000, client.id=atlas, security.protocol=PLAINTEXT}

cat atlas.20170512-081144.err

log4j:WARN Continuable parsing error 37 and column 14

log4j:WARN The content of element type "appender" must match "(errorHandler?,param*,rollingPolicy?,triggeringPolicy?,connectionSource?,layout?,filter*,appender-ref*)".

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

[#4 · Accepted solution]

@Saurabh

Can you attach atlas application.log file.

[#5]

Hello @Nixon Rodrigues,

I checked application.log and found following error

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.apache.atlas.web.dao.UserDao org.apache.atlas.web.service.UserService.userDao; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userDao': Invocation of init method failed; nested exception is java.lang.RuntimeException: org.apache.atlas.AtlasException: /usr/hdp/current/atlas-server/conf/users-credentials.properties not found in file system or as class loader resource

and /usr/hdp/current/atlas-server/conf/policy-store.txt not found in file system or as class loader resource.

So actually these files were not present, I don't know whether its a behavior of Atlas that we manually have to create them and update accordingly or it was just specific to my env due to some issue.

But once I have created and updated these two files then atlas started working.

[#6]

Hi Team.

Please find the below is application log details. Please do needful on this issue

2021-04-08 11:16:27,499 INFO - [main:] ~ Loading atlas-application.properties from file:/root/apache-atlas-2.0.0/conf/atlas-application.properties (ApplicationProperties:123)
2021-04-08 11:16:27,509 INFO - [main:] ~ Using graphdb backend 'janus' (ApplicationProperties:273)
2021-04-08 11:16:27,510 INFO - [main:] ~ Using storage backend 'hbase2' (ApplicationProperties:284)
2021-04-08 11:16:27,510 INFO - [main:] ~ Using index backend 'solr' (ApplicationProperties:295)
2021-04-08 11:16:27,513 INFO - [main:] ~ Setting solr-wait-searcher property 'true' (ApplicationProperties:301)
2021-04-08 11:16:27,513 INFO - [main:] ~ Setting index.search.map-name property 'false' (ApplicationProperties:305)
2021-04-08 11:16:27,513 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache = true (ApplicationProperties:318)
2021-04-08 11:16:27,513 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20 (ApplicationProperties:318)
2021-04-08 11:16:27,513 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache-size = 0.5 (ApplicationProperties:318)
2021-04-08 11:16:27,514 INFO - [main:] ~ Property (set to default) atlas.graph.cache.tx-cache-size = 15000 (ApplicationProperties:318)
2021-04-08 11:16:27,514 INFO - [main:] ~ Property (set to default) atlas.graph.cache.tx-dirty-size = 120 (ApplicationProperties:318)
2021-04-08 11:16:27,537 INFO - [main:] ~
########################################################################################
Atlas Server (STARTUP)

project.name: apache-atlas
project.description: Metadata Management and Data Governance Platform over Hadoop
build.user: root
build.epoch: 1617878818196
project.version: 2.0.0
build.version: 2.0.0
vc.revision: release
vc.source.url: scm:git:git://git.apache.org/atlas.git/atlas-webapp
######################################################################################## (Atlas:215)
2021-04-08 11:16:27,537 INFO - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:216)
2021-04-08 11:16:27,537 INFO - [main:] ~ Server starting with TLS ? false on port 21000 (Atlas:217)
2021-04-08 11:16:27,537 INFO - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:218)
2021-04-08 11:16:28,445 INFO - [main:] ~ No authentication method configured. Defaulting to simple authentication (LoginProcessor:102)
2021-04-08 11:16:28,591 WARN - [main:] ~ Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (NativeCodeLoader:60)
2021-04-08 11:16:28,618 INFO - [main:] ~ Logged in user root (auth:SIMPLE) (LoginProcessor:77)
2021-04-08 11:16:29,102 INFO - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189)
2021-04-08 11:16:53,718 WARN - [main:] ~ org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config' (Krb5HttpClientBuilder:142)
2021-04-08 11:16:54,047 INFO - [main:] ~ Failed to obtain graph instance, retrying 3 times, error: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.solr.Solr6Index (AtlasGraphProvider:100)
2021-04-08 11:17:24,369 WARN - [main:] ~ org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config' (Krb5HttpClientBuilder:142)
2021-04-08 11:17:24,379 WARN - [main:] ~ Failed to obtain graph instance on attempt 1 of 3 (AtlasGraphProvider:118)
java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.solr.Solr6Index
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:476)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:463)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:148)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1840)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:138)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)
at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:165)
at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraph(AtlasJanusGraphDatabase.java:263)
at org.apache.atlas.repository.graph.AtlasGraphProvider.getGraphInstance(AtlasGraphProvider.java:52)
at org.apache.atlas.repository.graph.AtlasGraphProvider.retry(AtlasGraphProvider.java:114)
at org.apache.atlas.repository.graph.AtlasGraphProvider.get(AtlasGraphProvider.java:102)
at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$ddd7ec69.CGLIB$get$1(<generated>)
at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$ddd7ec69$$FastClassBySpringCGLIB$$35af08e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$ddd7ec69.get(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1181)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1075)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.aop.framework.autoproxy.BeanFactoryAdvisorRetrievalHelper.findAdvisorBeans(BeanFactoryAdvisorRetrievalHelper.java:92)
at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.findCandidateAdvisors(AbstractAdvisorAutoProxyCreator.java:102)
at org.springframework.aop.aspectj.autoproxy.AspectJAwareAdvisorAutoProxyCreator.shouldSkip(AspectJAwareAdvisorAutoProxyCreator.java:103)
at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.postProcessBeforeInstantiation(AbstractAutoProxyCreator.java:248)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1045)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1019)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:473)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.apache.atlas.web.setup.KerberosAwareListener.contextInitialized(KerberosAwareListener.java:31)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:843)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:816)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)

[#7]

@Saurabh, these two files are needed by Atlas for authentication. I think files were not copied to atlas conf properly when you upgrade HDP.