Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

Apache atlas [import-hive.sh failed]

avatar
Rising Star

Atlas error while running import-hive.sh

Anycan help me..complete stack trace as below.

2016-10-20 19:23:36,626 DEBUG - [main:] ~ Using resource http://ServerName:21000/api/atlas/entities/4dc94048-3f95-42f7-ae5f-6e9b5baea06d for 0 times (AtlasClient:784) 2016-10-20 19:24:36,673 WARN - [main:] ~ Handled exception in calling api api/atlas/entities (AtlasClient:791) com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623) at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188) at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785) at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 14 more 2016-10-20 19:24:36,675 WARN - [main:] ~ Exception's cause: class java.net.SocketTimeoutException (AtlasClient:792) Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623) at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188) at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785) at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 14 more Failed to import Hive Data Model!!!

1 ACCEPTED SOLUTION

avatar
New Member

Starting Kafka fixed this issue for me!

View solution in original post

5 REPLIES 5

avatar
Rising Star

@Ayub Khan

I have seen you have resolved similar kind of problem with Atlas, could you please help me on the same

i used some references

http://atlas.incubator.apache.org/0.7.0-incubating/Bridge-Hive.html

https://community.hortonworks.com/questions/39839/how-to-import-metadata-from-hive-into-atlas-and-th...

Thanks in advance

Bhupesh Khanna

avatar
Reader

Hello,I met the same question as yours, but I really dont understand what the links say in your posts, could I get your explaination? just add the value of atlas.rest.address can solve this problem? But I have already add atlas.rest.address = <Atlas-host>:<port> property to atlas-application.properties,By the way, I am using atlas 1.2.0

Here is my atlas-application.properties:

atlas.graph.storage.backend=hbase
atlas.graph.storage.hbase.table=apache_atlas_janus


atlas.graph.storage.hostname=gaoenwei66.cmp3481.interconnect,gaoenwei67.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect
atlas.graph.storage.hbase.regions-per-server=1
atlas.graph.storage.lock.wait-time=10000

atlas.EntityAuditRepository.impl=org.apache.atlas.repository.audit.HBaseBasedAuditRepository

atlas.graph.index.search.backend=elasticsearch
atlas.graph.index.search.index-name=es_janusgraph

atlas.graph.index.search.hostname=gaoenwei70.cmp3481.interconnect,gaoenwei68.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect
atlas.graph.index.search.elasticsearch.client-only=true

atlas.graph.index.search.max-result-set-size=150

atlas.notification.embedded=false
#atlas.kafka.data=${sys:atlas.home}/data/kafka
atlas.kafka.zookeeper.connect=gaoenwei67.cmp3481.interconnect:2181,gaoenwei66.cmp3481.interconnect:2181,gaoenwei65.cmp3481.interconnect:2181
atlas.kafka.bootstrap.servers=gaoenwei67.cmp3481.interconnect:2181,gaoenwei66.cmp3481.interconnect:2181,gaoenwei65.cmp3481.interconnect:9092
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.connection.timeout.ms=200
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.auto.commit.interval.ms=1000
atlas.kafka.hook.group.id=atlas

atlas.kafka.enable.auto.commit=false
atlas.kafka.auto.offset.reset=earliest
atlas.kafka.session.timeout.ms=30000
atlas.kafka.offsets.topic.replication.factor=2
atlas.kafka.poll.timeout.ms=1000

atlas.notification.hook.topic.name=ATLAS_HOOK
atlas.notification.entities.topic.name=ATLAS_ENTITIES
 
atlas.notification.hook.consumer.topic.names=ATLAS_HOOK
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES

atlas.notification.create.topics=false
atlas.notification.replicas=2
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.notification.log.failed.messages=true
atlas.notification.consumer.retry.interval=500
atlas.notification.hook.retry.interval=1000

atlas.enableTLS=false

atlas.authentication.method.kerberos=false
atlas.authentication.method.file=true

#### ldap.type= LDAP or AD
atlas.authentication.method.ldap.type=none

#### user credentials file
atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties

atlas.rest.address=http://172.21.42.78:21000/
# If enabled and set to true, this will run setup steps when the server starts
atlas.server.run.setup.on.start=false

atlas.audit.hbase.tablename=apache_atlas_entity_audit
atlas.audit.zookeeper.session.timeout.ms=1000
atlas.audit.hbase.zookeeper.quorum=gaoenwei66.cmp3481.interconnect,gaoenwei67.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect:2181

atlas.server.ha.enabled=false

atlas.authorizer.impl=simple
atlas.authorizer.simple.authz.policy.file=atlas-simple-authz-policy.json

atlas.rest-csrf.enabled=true
atlas.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*,^Chrome.*
atlas.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE
atlas.rest-csrf.custom-header=X-XSRF-HEADER

atlas.metric.query.cache.ttlInSecs=900
atlas.search.gremlin.enable=false

atlas.hook.hive.synchronous=false
atlas.hook.hive.numRetries=3
atlas.hook.hive.queueSize=10000
atlas.cluster.name=primary
 
atlas.hook.hbase.synchronous=false
atlas.hook.hbase.numRetries=3 
atlas.hook.hbase.queueSize=10000 
atlas.cluster.name=primary

 

I really need your help, wish for your answer 

avatar
@Bhupesh Khanna

Please provide more details about your cluster. Is your cluster kerberized? Is ranger enabled? which version of HDP you have installed?

avatar
Rising Star

Hi @Ayub Khan

Below are the answers

Is your cluster kerberized? - No

Is ranger enabled? - Yes

which version of HDP you have installed? - HDP-2.5.0.0

please let me know if more information is required.

Thanks,

Bhupesh Khanna

avatar
New Member

Starting Kafka fixed this issue for me!