Atlas UI not working on Docker HDP 2.5 Sandbox after container crash

[#1]

I'm using the Docker version of the HDP 2.5 Sandbox. The Atlas UI was working fine as I completed this tutorial http://hortonworks.com/hadoop-tutorial/tag-based-policies-atlas-ranger/ and this tutorial http://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/.

However my Docker container crashed unexpectedly while I was doing some things in Hive. I restarted my Sandbox and now the Atlas UI won't come back up. This is what I see:

Looking at /var/log/atlas/application.log I see this:

7) Error injecting constructor, org.apache.atlas.typesystem.exception.TypeExistsException: Cannot redefine type TaxonomyTerm
  at org.apache.atlas.services.DefaultMetadataService.<init>(DefaultMetadataService.java:117)
  at org.apache.atlas.services.DefaultMetadataService.class(DefaultMetadataService.java:93)
  while locating org.apache.atlas.services.DefaultMetadataService
  at org.apache.atlas.RepositoryMetadataModule.configure(RepositoryMetadataModule.java:83)
  while locating org.apache.atlas.services.MetadataService
    for parameter 0 at org.apache.atlas.web.resources.EntityResource.<init>(EntityResource.java:97)
  at org.apache.atlas.web.resources.EntityResource.class(EntityResource.java:80)
  while locating org.apache.atlas.web.resources.EntityResource
    for parameter 1 at org.apache.atlas.LocalAtlasClient.<init>(LocalAtlasClient.java:55)
  while locating org.apache.atlas.LocalAtlasClient
    for parameter 1 at org.apache.atlas.notification.NotificationHookConsumer.<init>(NotificationHookConsumer.java:72)
  at org.apache.atlas.notification.NotificationHookConsumer.class(NotificationHookConsumer.java:48)
  while locating org.apache.atlas.notification.NotificationHookConsumer
Caused by: org.apache.atlas.typesystem.exception.TypeExistsException: Cannot redefine type TaxonomyTerm
    at org.apache.atlas.typesystem.types.TypeSystem$TransientTypeSystem.validateAndSetupShallowTypes(TypeSystem.java:428)
    at org.apache.atlas.typesystem.types.TypeSystem$TransientTypeSystem.verifyTypes(TypeSystem.java:705)
    at org.apache.atlas.typesystem.types.TypeSystem$TransientTypeSystem.defineTypes(TypeSystem.java:622)
    at org.apache.atlas.typesystem.types.TypeSystem.defineTypes(TypeSystem.java:284)
    at org.apache.atlas.typesystem.types.TypeSystem.defineTypes(TypeSystem.java:276)
    at org.apache.atlas.services.DefaultMetadataService.restoreTypeSystem(DefaultMetadataService.java:163)
    at org.apache.atlas.services.DefaultMetadataService.<init>(DefaultMetadataService.java:152)
    at org.apache.atlas.services.DefaultMetadataService.<init>(DefaultMetadataService.java:117)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at com.google.inject.internal.DefaultConstructionProxyFactory$2.newInstance(DefaultConstructionProxyFactory.java:86)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:96)
    at com.google.inject.Guice.createInjector(Guice.java:84)
    at org.apache.atlas.web.listeners.GuiceServletConfig.getInjector(GuiceServletConfig.java:77)
    at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:47)
    at org.apache.atlas.web.listeners.GuiceServletConfig.contextInitialized(GuiceServletConfig.java:131)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:387)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:354)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:93)
    at org.apache.atlas.Atlas.main(Atlas.java:113)

7 errors
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:96)
    at com.google.inject.Guice.createInjector(Guice.java:84)
    at org.apache.atlas.web.listeners.GuiceServletConfig.getInjector(GuiceServletConfig.java:77)
    at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:47)
    at org.apache.atlas.web.listeners.GuiceServletConfig.contextInitialized(GuiceServletConfig.java:131)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
    at org.eclipse.jetty.server.Server.start(Server.java:387)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
    at org.eclipse.jetty.server.Server.doStart(Server.java:354)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:93)
    at org.apache.atlas.Atlas.main(Atlas.java:113)

I have ensured all of the dependent services of Atlas are running (Ambari Infra, Storm, Kafka, etc).

I started a brand new container and I'm seeing the same behavior on a clean install, so there must be something I'm missing.

Any ideas?

[#2 · Accepted solution]

@Michael Young

To remove all the types/metadata from metadata store, please follow hte below steps..

• Stop Atlas
• Disable hbase table 'atlas_titan' - Hope you dont have any metadata there..
• Drop hbase table 'atlas_titan'
• Restart Atlas

This should actually remove all the existing types from backend store(hbase).

[#3]

@Ayub Pathan

You are a life saver! This worked like a charm!

[#4]

@Ayub Khan

Can you please tell how to disable hbase table "atlas_titan".

I am using ambari for managing services.

[#5]

@Ashis Panigrahi You need run "disable 'atlas_titan'" command in the hbase terminal to disable to hbase table.

[#6]

Note: Dropping the hbase table results in loosing the atlas data only drop if it is a fresh installation.

Actual issue is fixed in ATLAS-1542

[#7]

You have to login to HBase and remove master table atlas_titan as below

And restart service.

hbase(main):003:0> list

TABLE

ATLAS_ENTITY_AUDIT_EVENTS

atlas_titan

2 row(s) in 0.0070 seconds

=> ["ATLAS_ENTITY_AUDIT_EVENTS", "atlas_titan"]

hbase(main):005:0> disable 'atlas_titan'

0 row(s) in 2.5060 seconds

hbase(main):006:0> drop 'atlas_titan'

0 row(s) in 1.2730 seconds

hbase(main):007:0> exit

Restart Atlas service from Ambari UI