Support Questions
Find answers, ask questions, and share your expertise

Atlas can't see Hive tables

Atlas can't see Hive tables

Contributor

I'm following the instruction from here

http://hortonworks.com/hadoop-tutorial/tag-based-policies-atlas-ranger/

But I'm not using the sandbox, but a Ambari-installed HDP (Ambari v2.4.1.0, HDP v2.5.0). All services seems to be working fine according to Ambari. Creating tags in Atlas UI works but when I create a Hive table named "person", I can't search it in Atlas (in the tutorial the table is "employee").

22 REPLIES 22

Re: Atlas can't see Hive tables

@J. D. Bacolod

Have you created the hive table after installing the Atlas service? or before?

If it is before installing the Atlas service, You'd need to run import-hive.sh script https://community.hortonworks.com/articles/61274/import-hive-metadata-into-atlas.html and then try searching for the table in Atlas UI.

Re: Atlas can't see Hive tables

Contributor

I have the Hive table after the Atlas installation. I followed your steps in the link you provided, but I've got this:

Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log
2016-11-25 14:31:15,177 INFO  - [main:] ~ Looking for atlas-application.properties in classpath (ApplicationProperties:73)
2016-11-25 14:31:15,185 INFO  - [main:] ~ Loading atlas-application.properties from file:/etc/hive/2.5.0.0-1245/0/atlas-application.properties (ApplicationProperties:86)
2016-11-25 14:31:15,270 DEBUG - [main:] ~ Configuration loaded: (ApplicationProperties:99)
2016-11-25 14:31:15,270 DEBUG - [main:] ~ atlas.authentication.method.kerberos = False (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.cluster.name = <my-cluster-name> (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.keepAliveTime = 10 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.maxThreads = 5 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.minThreads = 5 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.numRetries = 3 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.queueSize = 1000 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.synchronous = true (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.bootstrap.servers = <my-single-node-hostname>:6667 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.hook.group.id = atlas (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.zookeeper.connect = <my-single-node-hostname>:2181 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.zookeeper.connection.timeout.ms = 200 (ApplicationProperties:102)
2016-11-25 14:31:15,274 DEBUG - [main:] ~ atlas.kafka.zookeeper.session.timeout.ms = 400 (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.kafka.zookeeper.sync.time.ms = 20 (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.notification.create.topics = True (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.notification.replicas = 1 (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.notification.topics = [ATLAS_HOOK, ATLAS_ENTITIES] (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.rest.address = http://<my-single-node-hostname>:21000 (ApplicationProperties:102)
2016-11-25 14:31:15,277 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:168)
2016-11-25 14:31:15,280 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:181)
2016-11-25 14:31:15,286 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:220)
2016-11-25 14:31:15,287 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:347)
2016-11-25 14:31:15,287 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:190)
2016-11-25 14:31:15,287 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:177)
Enter username for atlas :-
admin
Enter password for atlas :-
admin
2016-11-25 14:31:22,776 INFO  - [main:] ~ Client has only one service URL, will use that for all actions: http://<my-single-node-hostname>:21000 (AtlasClient:265)
2016-11-25 14:31:23,821 WARN  - [main:] ~ Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (NativeCodeLoader:62)
2016-11-25 14:31:24,960 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/types/hdfs_path for 0 times (AtlasClient:784)
2016-11-25 14:31:25,091 DEBUG - [main:] ~ API http://<my-single-node-hostname>:21000/api/atlas/types/hdfs_path returned status 200 (AtlasClient:1191)
2016-11-25 14:31:25,880 INFO  - [main:] ~ HDFS data model is already registered! (HiveMetaStoreBridge:609)
2016-11-25 14:31:25,881 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/types/hive_process for 0 times (AtlasClient:784)
2016-11-25 14:31:25,896 DEBUG - [main:] ~ API http://<my-single-node-hostname>:21000/api/atlas/types/hive_process returned status 200 (AtlasClient:1191)
2016-11-25 14:31:25,929 INFO  - [main:] ~ Hive data model is already registered! (HiveMetaStoreBridge:624)
2016-11-25 14:31:25,930 INFO  - [main:] ~ Importing hive metadata (HiveMetaStoreBridge:117)
2016-11-25 14:31:25,933 DEBUG - [main:] ~ Getting reference for database default (HiveMetaStoreBridge:211)
2016-11-25 14:31:25,934 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/entities?type=hive_db&property=qualifiedName&value=...; for 0 times (AtlasClient:784)
2016-11-25 14:31:26,185 DEBUG - [main:] ~ API http://<my-single-node-hostname>:21000/api/atlas/entities?type=hive_db&property=qualifiedName&value=...; returned status 404 (AtlasClient:1191)
2016-11-25 14:31:27,086 INFO  - [main:] ~ Importing objects from databaseName : default (HiveMetaStoreBridge:166)
2016-11-25 14:31:27,147 DEBUG - [main:] ~ creating instance of type hive_db (HiveMetaStoreBridge:193)
2016-11-25 14:31:27,836 DEBUG - [main:] ~ Submitting new entity hive_db = {
  "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
  "id":{
    "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
    "id":"-602949146493039",
    "version":0,
    "typeName":"hive_db",
    "state":"ACTIVE"
  },
  "typeName":"hive_db",
  "values":{
    "name":"default",
    "location":"hdfs://<my-single-node-hostname>:8020/apps/hive/warehouse",
    "description":"Default Hive database",
    "ownerType":2,
    "qualifiedName":"default@<my-cluster-name>",
    "owner":"public",
    "clusterName":"<my-cluster-name>",
    "parameters":{


    }
  },
  "traitNames":[


  ],
  "traits":{


  }
} (HiveMetaStoreBridge:196)
2016-11-25 14:31:27,836 DEBUG - [main:] ~ Creating entities: ["{\n  \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference\",\n  \"id\":{\n    \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Id\",\n    \"id\":\"-602949146493039\",\n    \"version\":0,\n    \"typeName\":\"hive_db\",\n    \"state\":\"ACTIVE\"\n  },\n  \"typeName\":\"hive_db\",\n  \"values\":{\n    \"name\":\"default\",\n    \"location\":\"hdfs:\/\/<my-single-node-hostname>:8020\/apps\/hive\/warehouse\",\n    \"description\":\"Default Hive database\",\n    \"ownerType\":2,\n    \"qualifiedName\":\"default@<my-cluster-name>\",\n    \"owner\":\"public\",\n    \"clusterName\":\"<my-cluster-name>\",\n    \"parameters\":{\n      \n    }\n  },\n  \"traitNames\":[\n    \n  ],\n  \"traits\":{\n    \n  }\n}"] (AtlasClient:694)
2016-11-25 14:31:27,836 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/entities for 0 times (AtlasClient:784)
2016-11-25 14:32:27,901 WARN  - [main:] ~ Handled exception in calling api api/atlas/entities (AtlasClient:791)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
        at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
        at com.sun.jersey.api.client.Client.handle(Client.java:648)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
        at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
        at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
        at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:695)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:712)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:197)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:155)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
        ... 15 more
2016-11-25 14:32:27,903 WARN  - [main:] ~ Exception's cause: class java.net.SocketTimeoutException (AtlasClient:792)
Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
        at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
        at com.sun.jersey.api.client.Client.handle(Client.java:648)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
        at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
        at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
        at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:695)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:712)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:197)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:155)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
        ... 15 more
Failed to import Hive Data Model!!!

Re: Atlas can't see Hive tables

Contributor

From the logs it looks like the end point

http://<my-single-node-hostname>:21000/api/atlas/entities

is not accessible

Is the service running ?

Re: Atlas can't see Hive tables

Contributor

Yes, that REST endpoint is working. When I

$ curl http://user:pwd@host:21000/api/atlas/entities?type=hive_db

I get

{"requestId":"qtp1635682869-21949 - 108fff64-d174-49b4-b822-8873456bf0a8","typeName":"hive_db","results":["7cd9c1e4-fc9a-4d8f-bd6b-d65d240fb355"],"count":1}

Re: Atlas can't see Hive tables

@J. D. Bacolod

Can you please check if the atlas service is running ? and if you are able to access http://<my-single-node-hostname>:21000/api/atlas/entities

Re: Atlas can't see Hive tables

Expert Contributor

@J. D. Bacolod

The atlas.rest.address property is not set properly.

edit the /etc/hive/conf/atlas-application.properties file and add a property

atlas.rest.address=http://<atlas server ip>:21000

Start the import-hive.sh script after making above change.

Re: Atlas can't see Hive tables

Contributor

That thing <my-single-node-hostname> is actually a Fully Qualified Domain Name (FQDN), I just masked it to hide my actual domain.

Re: Atlas can't see Hive tables

Expert Contributor

@J. D. Bacolod

Can you restart Kafka and try running import hive script.

Re: Atlas can't see Hive tables

Explorer

Hi, all my service are up and i when i tried to import hive script i found this errors /usr/bin/java and/or not found on the system. Please make sure java and jar commands are available.

Any suggestion please. Thx