Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Atlas can't see Hive tables

Highlighted

Atlas can't see Hive tables

Contributor

I'm following the instruction from here

http://hortonworks.com/hadoop-tutorial/tag-based-policies-atlas-ranger/

But I'm not using the sandbox, but a Ambari-installed HDP (Ambari v2.4.1.0, HDP v2.5.0). All services seems to be working fine according to Ambari. Creating tags in Atlas UI works but when I create a Hive table named "person", I can't search it in Atlas (in the tutorial the table is "employee").

22 REPLIES 22
Highlighted

Re: Atlas can't see Hive tables

@J. D. Bacolod

Have you created the hive table after installing the Atlas service? or before?

If it is before installing the Atlas service, You'd need to run import-hive.sh script https://community.hortonworks.com/articles/61274/import-hive-metadata-into-atlas.html and then try searching for the table in Atlas UI.

Highlighted

Re: Atlas can't see Hive tables

Contributor

I have the Hive table after the Atlas installation. I followed your steps in the link you provided, but I've got this:

Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/current/atlas-server/logs/import-hive.log
2016-11-25 14:31:15,177 INFO  - [main:] ~ Looking for atlas-application.properties in classpath (ApplicationProperties:73)
2016-11-25 14:31:15,185 INFO  - [main:] ~ Loading atlas-application.properties from file:/etc/hive/2.5.0.0-1245/0/atlas-application.properties (ApplicationProperties:86)
2016-11-25 14:31:15,270 DEBUG - [main:] ~ Configuration loaded: (ApplicationProperties:99)
2016-11-25 14:31:15,270 DEBUG - [main:] ~ atlas.authentication.method.kerberos = False (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.cluster.name = <my-cluster-name> (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.keepAliveTime = 10 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.maxThreads = 5 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.minThreads = 5 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.numRetries = 3 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.queueSize = 1000 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.hook.hive.synchronous = true (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.bootstrap.servers = <my-single-node-hostname>:6667 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.hook.group.id = atlas (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.zookeeper.connect = <my-single-node-hostname>:2181 (ApplicationProperties:102)
2016-11-25 14:31:15,273 DEBUG - [main:] ~ atlas.kafka.zookeeper.connection.timeout.ms = 200 (ApplicationProperties:102)
2016-11-25 14:31:15,274 DEBUG - [main:] ~ atlas.kafka.zookeeper.session.timeout.ms = 400 (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.kafka.zookeeper.sync.time.ms = 20 (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.notification.create.topics = True (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.notification.replicas = 1 (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.notification.topics = [ATLAS_HOOK, ATLAS_ENTITIES] (ApplicationProperties:102)
2016-11-25 14:31:15,276 DEBUG - [main:] ~ atlas.rest.address = http://<my-single-node-hostname>:21000 (ApplicationProperties:102)
2016-11-25 14:31:15,277 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:168)
2016-11-25 14:31:15,280 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:181)
2016-11-25 14:31:15,286 DEBUG - [main:] ~ ==> InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:220)
2016-11-25 14:31:15,287 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.initialize() (InMemoryJAASConfiguration:347)
2016-11-25 14:31:15,287 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:190)
2016-11-25 14:31:15,287 DEBUG - [main:] ~ <== InMemoryJAASConfiguration.init() (InMemoryJAASConfiguration:177)
Enter username for atlas :-
admin
Enter password for atlas :-
admin
2016-11-25 14:31:22,776 INFO  - [main:] ~ Client has only one service URL, will use that for all actions: http://<my-single-node-hostname>:21000 (AtlasClient:265)
2016-11-25 14:31:23,821 WARN  - [main:] ~ Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (NativeCodeLoader:62)
2016-11-25 14:31:24,960 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/types/hdfs_path for 0 times (AtlasClient:784)
2016-11-25 14:31:25,091 DEBUG - [main:] ~ API http://<my-single-node-hostname>:21000/api/atlas/types/hdfs_path returned status 200 (AtlasClient:1191)
2016-11-25 14:31:25,880 INFO  - [main:] ~ HDFS data model is already registered! (HiveMetaStoreBridge:609)
2016-11-25 14:31:25,881 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/types/hive_process for 0 times (AtlasClient:784)
2016-11-25 14:31:25,896 DEBUG - [main:] ~ API http://<my-single-node-hostname>:21000/api/atlas/types/hive_process returned status 200 (AtlasClient:1191)
2016-11-25 14:31:25,929 INFO  - [main:] ~ Hive data model is already registered! (HiveMetaStoreBridge:624)
2016-11-25 14:31:25,930 INFO  - [main:] ~ Importing hive metadata (HiveMetaStoreBridge:117)
2016-11-25 14:31:25,933 DEBUG - [main:] ~ Getting reference for database default (HiveMetaStoreBridge:211)
2016-11-25 14:31:25,934 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/entities?type=hive_db&property=qualifiedName&value=...; for 0 times (AtlasClient:784)
2016-11-25 14:31:26,185 DEBUG - [main:] ~ API http://<my-single-node-hostname>:21000/api/atlas/entities?type=hive_db&property=qualifiedName&value=...; returned status 404 (AtlasClient:1191)
2016-11-25 14:31:27,086 INFO  - [main:] ~ Importing objects from databaseName : default (HiveMetaStoreBridge:166)
2016-11-25 14:31:27,147 DEBUG - [main:] ~ creating instance of type hive_db (HiveMetaStoreBridge:193)
2016-11-25 14:31:27,836 DEBUG - [main:] ~ Submitting new entity hive_db = {
  "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
  "id":{
    "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
    "id":"-602949146493039",
    "version":0,
    "typeName":"hive_db",
    "state":"ACTIVE"
  },
  "typeName":"hive_db",
  "values":{
    "name":"default",
    "location":"hdfs://<my-single-node-hostname>:8020/apps/hive/warehouse",
    "description":"Default Hive database",
    "ownerType":2,
    "qualifiedName":"default@<my-cluster-name>",
    "owner":"public",
    "clusterName":"<my-cluster-name>",
    "parameters":{


    }
  },
  "traitNames":[


  ],
  "traits":{


  }
} (HiveMetaStoreBridge:196)
2016-11-25 14:31:27,836 DEBUG - [main:] ~ Creating entities: ["{\n  \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference\",\n  \"id\":{\n    \"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Id\",\n    \"id\":\"-602949146493039\",\n    \"version\":0,\n    \"typeName\":\"hive_db\",\n    \"state\":\"ACTIVE\"\n  },\n  \"typeName\":\"hive_db\",\n  \"values\":{\n    \"name\":\"default\",\n    \"location\":\"hdfs:\/\/<my-single-node-hostname>:8020\/apps\/hive\/warehouse\",\n    \"description\":\"Default Hive database\",\n    \"ownerType\":2,\n    \"qualifiedName\":\"default@<my-cluster-name>\",\n    \"owner\":\"public\",\n    \"clusterName\":\"<my-cluster-name>\",\n    \"parameters\":{\n      \n    }\n  },\n  \"traitNames\":[\n    \n  ],\n  \"traits\":{\n    \n  }\n}"] (AtlasClient:694)
2016-11-25 14:31:27,836 DEBUG - [main:] ~ Using resource http://<my-single-node-hostname>:21000/api/atlas/entities for 0 times (AtlasClient:784)
2016-11-25 14:32:27,901 WARN  - [main:] ~ Handled exception in calling api api/atlas/entities (AtlasClient:791)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
        at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
        at com.sun.jersey.api.client.Client.handle(Client.java:648)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
        at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
        at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
        at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:695)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:712)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:197)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:155)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
        ... 15 more
2016-11-25 14:32:27,903 WARN  - [main:] ~ Exception's cause: class java.net.SocketTimeoutException (AtlasClient:792)
Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
        at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
        at com.sun.jersey.api.client.Client.handle(Client.java:648)
        at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
        at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
        at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
        at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)
        at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)
        at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:695)
        at org.apache.atlas.AtlasClient.createEntity(AtlasClient.java:712)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:197)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:155)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)
        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:152)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1324)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
        at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
        ... 15 more
Failed to import Hive Data Model!!!
Highlighted

Re: Atlas can't see Hive tables

Contributor

From the logs it looks like the end point

http://<my-single-node-hostname>:21000/api/atlas/entities

is not accessible

Is the service running ?

Highlighted

Re: Atlas can't see Hive tables

Contributor

Yes, that REST endpoint is working. When I

$ curl http://user:pwd@host:21000/api/atlas/entities?type=hive_db

I get

{"requestId":"qtp1635682869-21949 - 108fff64-d174-49b4-b822-8873456bf0a8","typeName":"hive_db","results":["7cd9c1e4-fc9a-4d8f-bd6b-d65d240fb355"],"count":1}
Highlighted

Re: Atlas can't see Hive tables

@J. D. Bacolod

Can you please check if the atlas service is running ? and if you are able to access http://<my-single-node-hostname>:21000/api/atlas/entities

Highlighted

Re: Atlas can't see Hive tables

Expert Contributor

@J. D. Bacolod

The atlas.rest.address property is not set properly.

edit the /etc/hive/conf/atlas-application.properties file and add a property

atlas.rest.address=http://<atlas server ip>:21000

Start the import-hive.sh script after making above change.

Highlighted

Re: Atlas can't see Hive tables

Contributor

That thing <my-single-node-hostname> is actually a Fully Qualified Domain Name (FQDN), I just masked it to hide my actual domain.

Highlighted

Re: Atlas can't see Hive tables

Expert Contributor

@J. D. Bacolod

Can you restart Kafka and try running import hive script.

Highlighted

Re: Atlas can't see Hive tables

Explorer

Hi, all my service are up and i when i tried to import hive script i found this errors /usr/bin/java and/or not found on the system. Please make sure java and jar commands are available.

Any suggestion please. Thx

Don't have an account?
Coming from Hortonworks? Activate your account here