Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive tables not shown in atlas

Highlighted

Hive tables not shown in atlas

Explorer

Hi,

I'm trying to execute this command for importing data from hive to atlas:

/usr/hdp/current/atlas-server/hook-bin/import-hive.sh

But I'm stuck on this error

8860-screenshot-17.png

Thank you.

7 REPLIES 7
Highlighted

Re: Hive tables not shown in atlas

Expert Contributor

Have you set the environment variables $HADOOP_CLASSPATH and $HADOOP_HOME? If so, can you provide the values?

http://atlas.incubator.apache.org/Bridge-Hive.html

Highlighted

Re: Hive tables not shown in atlas

Explorer

Hi,

Here are details of my environment:

8925-2016-10-26.png

Thank you

Highlighted

Re: Hive tables not shown in atlas

Expert Contributor

It looks like the import-hive.sh script is not setting the classpath correctly. I've added some lines to the script that has gotten me past a number of NoClassDefFound issues:

    109 for i in "${HADOOP_HOME}/lib/"*.jar; do
    110         HADOOP_CP="${HADOOP_CP}:$i"
    111 done
    112 
    113 for i in "/usr/hdp/2.5.0.0-1245/hadoop/"*.jar; do
    114         HADOOP_CP="${HADOOP_CP}:$i"
    115 done
    140 "${JAVA_BIN}" ${JAVA_PROPERTIES} -cp "${CP}:/usr/hdp/2.5.0.0-1245/hadoop/client/xercesImpl.jar:/usr/hdp/2.5.0.0-1245/hadoop/client/jersey-client.jar:/usr/hdp/2.5.0.0-1245/hadoop-mapreduce/hadoop-mapreduce-client
        -core.jar" org.apache.atlas.hive.bridge.HiveMetaStoreBridge

This allows the command to begin running, but during the import, but it errors out with the following:

2016-10-27 19:34:17,844 DEBUG - [main:] ~ Using resource http://sandbox.hortonworks.com:21000/api/atlas/entities/b4099448-0871-4293-9aad-d03f8c8434c4 for 0 times (AtlasClient:784) 
2016-10-27 19:35:17,854 WARN  - [main:] ~ Handled exception in calling api api/atlas/entities (AtlasClient:791)                                                                       com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out                                                                                    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)                                                                        at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)                                                                                         at com.sun.jersey.api.client.Client.handle(Client.java:648)                                at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)                     at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)                  at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)             at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188)                at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785)                  at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214)                            at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808)                        at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506)                                                                                     at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159)                                                                                   at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124)                                                                                    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118)                                                                                at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662)    Caused by: java.net.SocketTimeoutException: Read timed out                                at java.net.SocketInputStream.socketRead0(Native Method)                                  at java.net.SocketInputStream.read(SocketInputStream.java:152)                            at java.net.SocketInputStream.read(SocketInputStream.java:122)                            at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)                         at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)                        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)                         at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)                       at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)                             at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1325)at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)                  at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)                                                                       at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)  

I'll keep at it.

Highlighted

Re: Hive tables not shown in atlas

Expert Contributor

Interestingly, I attempted to use curl to make the same call as it appears has failed in the above stack trace. And it worked (I changed atlas server address to localhost):

curl -u admin:admin -X GET http://localhost:21000/api/atlas/entities/b4099448-0871-4293-9aad-d03f8c8434c4 
{"requestId":"qtp940790097-16 - 409dcc45-2025-4c1d-bfd8-4dce1c414989","definition":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference","id":{"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id","id":"b4099448-0871-4293-9aad-d03f8c8434c4","version":0,"typeName":"hive_db","state":"ACTIVE"},"typeName":"hive_db","values":{"name":"default","location":"hdfs:\/\/sandbox.hortonworks.com:8020\/apps\/hive\/warehouse","description":"Default Hive database","ownerType":{"value":"ROLE","ordinal":2},"qualifiedName":"default@Sandbox","owner":"public","clusterName":"Sandbox","parameters":null},"traitNames":[],"traits":{}}}

Re: Hive tables not shown in atlas

Expert Contributor

Hey, Thanks for this response it helped me.

Highlighted

Re: Hive tables not shown in atlas

Guru

This sounds like an expirience I had with the sandbox. It turned out to be corrupted jar files that the import script depends on. Even after importing the jar files from a working environment and deleting the corrupt files new errors were revealed.

You might try getting fresh jar files to replace the current ones but I recommend starting from scratch with a new VM and always saving the machine state when you make any changes to the environment before shutting it down to avoid this issue.

Highlighted

Re: Hive tables not shown in atlas

New Contributor

Hi,

I got the same bug on a 2.6 sandbox with Atals 0.8. I opened a Jira and made a pull request. See https://issues.apache.org/jira/browse/ATLAS-1910

Regards

Don't have an account?
Coming from Hortonworks? Activate your account here