Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Atlas Metadata Server error HTTP 503 response from http://localhost:21000/api/atlas/admin/status in 0.000s (HTTP Error 503: Service Unavailable)

avatar
Guru

When I upgraded hdp from 2.5.3 to 2.6.0.3 then I am getting following warning in Ambari:

HTTP 503 response from http://localhost:21000/api/atlas/admin/status in 0.000s (HTTP Error 503: Service Unavailable)

Also following error in application.log file

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Menlo} span.s1 {font-variant-ligatures: no-common-ligatures} span.s2 {font-variant-ligatures: no-common-ligatures; background-color: #e6e600}

2017-05-12 00:06:43,877 INFO - [qtp2050835901-1065 - 0d5633f1-091f-47ac-949e-ed1f3583122e:] ~ Audit: UNKNOWN/127.0.0.1-127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:06Z (AUDIT:104)

2017-05-12 00:07:44,005 INFO - [qtp2050835901-1066 - 329b25a9-f9c9-4142-ae11-42f9faed4e7b:] ~ Audit: UNKNOWN/127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:07Z (AuditFilter:91)

2017-05-12 00:07:44,006 INFO - [qtp2050835901-1066 - 329b25a9-f9c9-4142-ae11-42f9faed4e7b:] ~ Audit: UNKNOWN/127.0.0.1-127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:07Z (AUDIT:104)

2017-05-12 00:08:43,872 INFO - [qtp2050835901-1066 - 2e95c961-12cc-4353-a360-32c1f40f6678:] ~ Audit: UNKNOWN/127.0.0.1 performed request GET http://localhost:21000/api/atlas/admin/status (127.0.0.1) at time 2017-05-12T04:08Z (AuditFilter:91)

1 ACCEPTED SOLUTION

avatar
Expert Contributor

@Saurabh

Can you attach atlas application.log file.

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@Saurabh

Please check if your Atlas Service is running fine on port 21000 on localhost?

# netstat -tnlpa | grep  21000

.

Also do you see any ERROR/ Warning on atlas log?

# less /var/log/atlas/atlas.20170419-185147.err
# less /var/log/atlas/atlas.20170419-185147.out 

.

avatar
Guru

I see following logs in those files.

netstat -tnlpa | grep 21000

(No info could be read for "-p": geteuid()=4569 but you should be root.)

tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN -

cat atlas.20170512-081144.out

{metadata.broker.list=<server>:6667, request.timeout.ms=30000, client.id=atlas, security.protocol=PLAINTEXT}

cat atlas.20170512-081144.err

log4j:WARN Continuable parsing error 37 and column 14

log4j:WARN The content of element type "appender" must match "(errorHandler?,param*,rollingPolicy?,triggeringPolicy?,connectionSource?,layout?,filter*,appender-ref*)".

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

avatar
Expert Contributor

@Saurabh

Can you attach atlas application.log file.

avatar
Guru

Hello @Nixon Rodrigues,

I checked application.log and found following error

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private org.apache.atlas.web.dao.UserDao org.apache.atlas.web.service.UserService.userDao; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userDao': Invocation of init method failed; nested exception is java.lang.RuntimeException: org.apache.atlas.AtlasException: /usr/hdp/current/atlas-server/conf/users-credentials.properties not found in file system or as class loader resource

and /usr/hdp/current/atlas-server/conf/policy-store.txt not found in file system or as class loader resource.

So actually these files were not present, I don't know whether its a behavior of Atlas that we manually have to create them and update accordingly or it was just specific to my env due to some issue.

But once I have created and updated these two files then atlas started working.

avatar
New Contributor

Hi Team.

 

Please find the below is application log details. Please do needful on this issue

2021-04-08 11:16:27,499 INFO - [main:] ~ Loading atlas-application.properties from file:/root/apache-atlas-2.0.0/conf/atlas-application.properties (ApplicationProperties:123)
2021-04-08 11:16:27,509 INFO - [main:] ~ Using graphdb backend 'janus' (ApplicationProperties:273)
2021-04-08 11:16:27,510 INFO - [main:] ~ Using storage backend 'hbase2' (ApplicationProperties:284)
2021-04-08 11:16:27,510 INFO - [main:] ~ Using index backend 'solr' (ApplicationProperties:295)
2021-04-08 11:16:27,513 INFO - [main:] ~ Setting solr-wait-searcher property 'true' (ApplicationProperties:301)
2021-04-08 11:16:27,513 INFO - [main:] ~ Setting index.search.map-name property 'false' (ApplicationProperties:305)
2021-04-08 11:16:27,513 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache = true (ApplicationProperties:318)
2021-04-08 11:16:27,513 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20 (ApplicationProperties:318)
2021-04-08 11:16:27,513 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache-size = 0.5 (ApplicationProperties:318)
2021-04-08 11:16:27,514 INFO - [main:] ~ Property (set to default) atlas.graph.cache.tx-cache-size = 15000 (ApplicationProperties:318)
2021-04-08 11:16:27,514 INFO - [main:] ~ Property (set to default) atlas.graph.cache.tx-dirty-size = 120 (ApplicationProperties:318)
2021-04-08 11:16:27,537 INFO - [main:] ~
########################################################################################
Atlas Server (STARTUP)

project.name: apache-atlas
project.description: Metadata Management and Data Governance Platform over Hadoop
build.user: root
build.epoch: 1617878818196
project.version: 2.0.0
build.version: 2.0.0
vc.revision: release
vc.source.url: scm:git:git://git.apache.org/atlas.git/atlas-webapp
######################################################################################## (Atlas:215)
2021-04-08 11:16:27,537 INFO - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:216)
2021-04-08 11:16:27,537 INFO - [main:] ~ Server starting with TLS ? false on port 21000 (Atlas:217)
2021-04-08 11:16:27,537 INFO - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:218)
2021-04-08 11:16:28,445 INFO - [main:] ~ No authentication method configured. Defaulting to simple authentication (LoginProcessor:102)
2021-04-08 11:16:28,591 WARN - [main:] ~ Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (NativeCodeLoader:60)
2021-04-08 11:16:28,618 INFO - [main:] ~ Logged in user root (auth:SIMPLE) (LoginProcessor:77)
2021-04-08 11:16:29,102 INFO - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189)
2021-04-08 11:16:53,718 WARN - [main:] ~ org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config' (Krb5HttpClientBuilder:142)
2021-04-08 11:16:54,047 INFO - [main:] ~ Failed to obtain graph instance, retrying 3 times, error: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.solr.Solr6Index (AtlasGraphProvider:100)
2021-04-08 11:17:24,369 WARN - [main:] ~ org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder is configured without specifying system property 'java.security.auth.login.config' (Krb5HttpClientBuilder:142)
2021-04-08 11:17:24,379 WARN - [main:] ~ Failed to obtain graph instance on attempt 1 of 3 (AtlasGraphProvider:118)
java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.solr.Solr6Index
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:64)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:476)
at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:463)
at org.janusgraph.diskstorage.Backend.<init>(Backend.java:148)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1840)
at org.janusgraph.graphdb.database.StandardJanusGraph.<init>(StandardJanusGraph.java:138)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111)
at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:165)
at org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraph(AtlasJanusGraphDatabase.java:263)
at org.apache.atlas.repository.graph.AtlasGraphProvider.getGraphInstance(AtlasGraphProvider.java:52)
at org.apache.atlas.repository.graph.AtlasGraphProvider.retry(AtlasGraphProvider.java:114)
at org.apache.atlas.repository.graph.AtlasGraphProvider.get(AtlasGraphProvider.java:102)
at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$ddd7ec69.CGLIB$get$1(<generated>)
at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$ddd7ec69$$FastClassBySpringCGLIB$$35af08e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
at org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$ddd7ec69.get(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1181)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1075)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1201)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1103)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.aop.framework.autoproxy.BeanFactoryAdvisorRetrievalHelper.findAdvisorBeans(BeanFactoryAdvisorRetrievalHelper.java:92)
at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.findCandidateAdvisors(AbstractAdvisorAutoProxyCreator.java:102)
at org.springframework.aop.aspectj.autoproxy.AspectJAwareAdvisorAutoProxyCreator.shouldSkip(AspectJAwareAdvisorAutoProxyCreator.java:103)
at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.postProcessBeforeInstantiation(AbstractAutoProxyCreator.java:248)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1045)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.resolveBeforeInstantiation(AbstractAutowireCapableBeanFactory.java:1019)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:473)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.apache.atlas.web.setup.KerberosAwareListener.contextInitialized(KerberosAwareListener.java:31)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:843)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:816)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)

 

avatar
Expert Contributor

@Saurabh, these two files are needed by Atlas for authentication. I think files were not copied to atlas conf properly when you upgrade HDP.