@mqureshi thank you for the reply. Do you happen to know any guide where the inner working of Atlas (related to indexing) is explained in detail? If yes, would you be kind enough to point me to it please? Thanking you in anticipation. Kind Regards Bilal ... View more

@mqureshi Thank you for the reply. I am using the following (link: http://atlas.apache.org/InstallationSteps.html) for Atlas installation. I can view the schema.xml under the conf folder within the atlas installation. How can we add indexes in Solr based on our data stored in Atlas? Is there a mechanism to do so, or like the schema.xml it is hidden from users? ... View more

@Graham Martin there is nothing wrong with the Atlas UI. I want to display the data in my Atlas in a graph based visualisation, not just the lineage. That is why I require a method a visualise data ingested into Atlas. I hope I can help you to understand the issue. ... View more

@Ayub Khan The atlas.20170227-122728.err contains the following error, I am running atlas from the hduser account so I am surprised why is it saying permission denied: log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /home/hduser/atlas/distro/target/apache-atlas-0.8-incubating-SNAPSHOT-bin/apache-atlas-0.8-incubating-SNAPSHOT/logs/application.log (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:142) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:295) at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:176) at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:191) at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:523) at org.apache.log4j.xml.DOMConfigurator.parseCategory(DOMConfigurator.java:436) at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1004) at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:872) at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:778) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66) at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72) at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45) at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at org.apache.atlas.Atlas.<clinit>(Atlas.java:43) log4j:ERROR Either File or DatePattern options are not set for appender [FILE]. The quick_start.log is showing the following: 017-02-23 10:04:38,562 INFO - [main:] ~ Looking for atlas-application.properties in classpath (ApplicationProperties:73) 2017-02-23 10:04:38,565 INFO - [main:] ~ Loading atlas-application.properties from file:/home/hduser/atlas/distro/target/apache-atlas-0.8-incubating-SNAPSHOT-bin/apache-atlas-0.8-incubating-SNAPSHOT/conf$ 2017-02-23 10:04:43,505 INFO - [main:] ~ Client has only one service URL, will use that for all actions: http://localhost:21000 (AtlasBaseClient:198) 2017-02-23 10:04:43,719 INFO - [main:] ~ Client has only one service URL, will use that for all actions: http://localhost:21000 (AtlasBaseClient:198) 2017-02-23 10:04:43,811 INFO - [main:] ~ Client has only one service URL, will use that for all actions: http://localhost:21000 (AtlasBaseClient:198) 2017-02-23 10:04:43,921 INFO - [main:] ~ Client has only one service URL, will use that for all actions: http://localhost:21000 (AtlasBaseClient:198) 2017-02-23 10:04:44,076 ERROR - [main:] ~ Got a service unavailable when calling: http://localhost:21000/api/atlas/v2/types/typedefs/, will retry.. (AtlasBaseClient:320) 2017-02-23 10:04:49,079 ERROR - [main:] ~ Got a service unavailable when calling: http://localhost:21000/api/atlas/v2/types/typedefs/, will retry.. (AtlasBaseClient:320) 2017-02-23 10:04:54,086 ERROR - [main:] ~ Got a service unavailable when calling: http://localhost:21000/api/atlas/v2/types/typedefs/, will retry.. (AtlasBaseClient:320) 2017-02-23 10:04:59,091 ERROR - [main:] ~ Got a service unavailable when calling: http://localhost:21000/api/atlas/v2/types/typedefs/, will retry.. (AtlasBaseClient:320) ... View more

@Ayub Khan Application.log contains the following: java.lang.RuntimeException: cannot use dir /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so at org.reflections.vfs.SystemDir.<init>(SystemDir.java:20) at org.reflections.vfs.Vfs$DefaultUrlTypes$3.createDir(Vfs.java:237) at org.reflections.vfs.Vfs.fromURL(Vfs.java:99) at org.reflections.vfs.Vfs.fromURL(Vfs.java:91) at org.reflections.Reflections.scan(Reflections.java:231) at org.reflections.Reflections.scan(Reflections.java:204) at org.reflections.Reflections.<init>(Reflections.java:129) 2017-02-23 09:24:24,583 WARN - [main:] ~ could not create Vfs.Dir from url. ignoring the exception and continuing (Reflections:208) org.reflections.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/ext/libatk-wrapper.so] either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType. at org.reflections.vfs.Vfs.fromURL(Vfs.java:109) at org.reflections.vfs.Vfs.fromURL(Vfs.java:91) at org.reflections.Reflections.scan(Reflections.java:231) at org.reflections.Reflections.scan(Reflections.java:204) at org.reflections.Reflections.<init>(Reflections.java:129) ... View more

@Ayub Khan Running bin/quick_start.py resulted in the following output: Enter username for atlas :- admin Enter password for atlas :- admin Creating sample types: Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasBaseClient$APIInfo@30432a9e failed with status 503 (Service Unavailable) Response Body (<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 503 </title> </head> <body> <h2>HTTP ERROR: 503</h2> <p>Problem accessing /api/atlas/v2/types/typedefs/. Reason: <pre> Service Unavailable</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html> ) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:327) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:419) at org.apache.atlas.AtlasTypedefClientV2.createAtlasTypeDefs(AtlasTypedefClientV2.java:147) at org.apache.atlas.examples.QuickStartV2.createTypes(QuickStartV2.java:206) at org.apache.atlas.examples.QuickStartV2.runQuickstart(QuickStartV2.java:153) at org.apache.atlas.examples.QuickStartV2.main(QuickStartV2.java:138) No sample data added to Apache Atlas Server. ... View more

@Ayub Khan I followed the steps as suggested above and managed to get the following results: hduser@ubuntu:~/atlas/distro/target/apache-atlas-0.8-incubating-SNAPSHOT-bin/apache-atlas-0.8-incubating-SNAPSHOT$ bin/atlas_start.py configured for local hbase. hbase started. configured for local solr. solr started. setting up solr collections... Apache Atlas Server started!!! However I am not able to access the dashboard. I tried running the command: curl -v http://localhost:21000/api/atlas/admin/version And it gave me the following output: hduser@ubuntu:~/atlas/distro/target/apache-atlas-0.8-incubating-SNAPSHOT-bin/apache-atlas-0.8-incubating-SNAPSHOT$ curl -v http://localhost:21000/api/atlas/admin/version * Hostname was NOT found in DNS cache * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 21000 (#0) > GET /api/atlas/admin/version HTTP/1.1 > User-Agent: curl/7.35.0 > Host: localhost:21000 > Accept: */* > < HTTP/1.1 503 Service Unavailable < Date: Thu, 23 Feb 2017 09:43:12 GMT < Cache-Control: must-revalidate,no-cache,no-store < Content-Type: text/html; charset=ISO-8859-1 < Content-Length: 311 * Server Jetty(9.2.12.v20150709) is not blacklisted < Server: Jetty(9.2.12.v20150709) < <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 503 </title> </head> <body> <h2>HTTP ERROR: 503</h2> <p>Problem accessing /api/atlas/admin/version. Reason: <pre> Service Unavailable</pre></p> <hr /><i><small>Powered by Jetty://</small></i> </body> </html> * Connection #0 to host localhost left intact Not sure what this means! :S The atlas-20170223-094455.err file contains the following: log4j:ERROR setFile(null,true) call failed. java.io.FileNotFoundException: /home/hduser/atlas/distro/target/apache-atlas-0.8-incubating-SNAPSHOT-bin/apache-atlas-0.8-incubating-SNAPSHOT/logs/application.log (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:142) at org.apache.log4j.FileAppender.setFile(FileAppender.java:294) at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165) at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223) at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307) at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:295) at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:176) at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:191) at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:523) at org.apache.log4j.xml.DOMConfigurator.parseCategory(DOMConfigurator.java:436) at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1004) at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:872) at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:778) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66) at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72) at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45) at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at org.apache.atlas.Atlas.<clinit>(Atlas.java:43) log4j:ERROR Either File or DatePattern options are not set for appender [FILE]. ... View more

@Ayub Khan Also I just happened to look at the atlas.20170222-161714.err file and following are the contents. I presume that the address Atlas needs is already in use? If this is the case how do I free it? Exception in thread "main" MultiException[java.lang.ExceptionInInitializerError, java.net.BindException: Address already in use] at org.eclipse.jetty.server.Server.doStart(Server.java:329) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:92) at org.apache.atlas.Atlas.main(Atlas.java:118) java.lang.ExceptionInInitializerError at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:195) at org.apache.atlas.ApplicationProperties.getClass(ApplicationProperties.java:120) at org.apache.atlas.util.AtlasRepositoryConfiguration.getDeleteHandlerImpl(AtlasRepositoryConfiguration.java:102) at org.apache.atlas.RepositoryMetadataModule.configure(RepositoryMetadataModule.java:109) at com.google.inject.AbstractModule.configure(AbstractModule.java:62) at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) at com.google.inject.spi.Elements.getElements(Elements.java:110) at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) at com.google.inject.Guice.createInjector(Guice.java:99) at com.google.inject.Guice.createInjector(Guice.java:84) at org.apache.atlas.web.listeners.GuiceServletConfig.getInjector(GuiceServletConfig.java:77) at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:47) at org.apache.atlas.web.listeners.GuiceServletConfig.contextInitialized(GuiceServletConfig.java:142) at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800) at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444) at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.server.Server.start(Server.java:387) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.Server.doStart(Server.java:354) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:92) at org.apache.atlas.Atlas.main(Atlas.java:118) Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.solr.Solr5Index at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:55) at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:421) at com.thinkaurelius.titan.diskstorage.Backend.getIndexes(Backend.java:408) at com.thinkaurelius.titan.diskstorage.Backend.<init>(Backend.java:139) at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1748) at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:110) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93) at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73) at org.apache.atlas.repository.graphdb.titan0.Titan0GraphDatabase.getGraphInstance(Titan0GraphDatabase.java:115) at org.apache.atlas.repository.graphdb.titan0.Titan0GraphDatabase.getGraph(Titan0GraphDatabase.java:171) at org.apache.atlas.repository.graph.AtlasGraphProvider.getGraphInstance(AtlasGraphProvider.java:38) at org.apache.atlas.repository.graph.GraphHelper.getInstance(GraphHelper.java:113) at org.apache.atlas.repository.graph.DeleteHandler.<clinit>(DeleteHandler.java:54) ... 32 more ... View more

@Nixon Rodrigues Running the above command is yielding the output attached, not sure what to take from it though! ... View more

@Ayub Khan Application.log states the following: 2017-02-22 16:02:14,716 WARN - [pool-1-thread-5:] ~ HttpConnection@1d3946db{FILLING} (HttpConnection:277) java.lang.IllegalStateException: s=DISPATCHED i=true a=null at org.eclipse.jetty.server.HttpChannelState.handling(HttpChannelState.java:232) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:282) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-02-22 16:02:17,506 WARN - [pool-1-thread-8:] ~ HttpConnection@a0fc6a6{FILLING} (HttpConnection:277) java.lang.IllegalStateException: s=DISPATCHED i=true a=null at org.eclipse.jetty.server.HttpChannelState.handling(HttpChannelState.java:232) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:282) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Not sure how to fix it? ... View more

@Ayub Khan I got the following upon running the command: netstat -planet | grep 21000 hduser@ubuntu:~/atlas/distro/target/apache-atlas-0.8-incubating-SNAPSHOT-bin/apache-atlas-0.8-incubating-SNAPSHOT$ netstat -planet | grep 21000 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN 1001 259958 15739/java tcp 0 0 127.0.0.1:21000 127.0.0.1:54856 TIME_WAIT 0 0 - ... View more

@Ayub Khan When I try checking its status with the command mentioned above I am getting the following: * Hostname was NOT found in DNS cache * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 21000 (#0) > GET /api/atlas/admin/version HTTP/1.1 > User-Agent: curl/7.35.0 > Host: localhost:21000 > Accept: */* > * Empty reply from server * Connection #0 to host localhost left intact curl: (52) Empty reply from server Trying to load quick start model is giving me the same exception as above. ... View more

@Ned Shawa I tried to follow the example above to import a csv file named drivers. hbase(main):001:0> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, –Dimporttsv.columns=”HBASE_ROW_KEY,driver_id,driver_name,certified,wage_plan” drivers /home/bilal/drivers.csv SyntaxError: (hbase):1: syntax error, unexpected ',' hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, ^ I am getting the following SyntaxError unexpected ','. Would you be kind enough to suggest the solution? Thanking you in anticipation. ... View more

@Ayub Khan any suggestions Sir? ... View more

Thank you @Ayub Khan, I am using an embedded hbase-solr configuration of Atlas so I presume I need to set the configurations manually. ... View more

@anaik thank you for the response. Would you be kind enough to provide a step by step guide on how to import metadata from mySQL database to Atlas if possible? ... View more

@Vadim Vaks would you be kind enough to provide the implementation details for the above? Thanking you in anticipation. ... View more