Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Apache atlas [import-hive.sh failed]

avatar
Rising Star

Atlas error while running import-hive.sh

Anycan help me..complete stack trace as below.

2016-10-20 19:23:36,626 DEBUG - [main:] ~ Using resource http://ServerName:21000/api/atlas/entities/4dc94048-3f95-42f7-ae5f-6e9b5baea06d for 0 times (AtlasClient:784) 2016-10-20 19:24:36,673 WARN - [main:] ~ Handled exception in calling api api/atlas/entities (AtlasClient:791) com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623) at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188) at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785) at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 14 more 2016-10-20 19:24:36,675 WARN - [main:] ~ Exception's cause: class java.net.SocketTimeoutException (AtlasClient:792) Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623) at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188) at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785) at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 14 more Failed to import Hive Data Model!!!

1 ACCEPTED SOLUTION

avatar

Starting Kafka fixed this issue for me!

View solution in original post

5 REPLIES 5

avatar
Rising Star

@Ayub Khan

I have seen you have resolved similar kind of problem with Atlas, could you please help me on the same

i used some references

http://atlas.incubator.apache.org/0.7.0-incubating/Bridge-Hive.html

https://community.hortonworks.com/questions/39839/how-to-import-metadata-from-hive-into-atlas-and-th...

Thanks in advance

Bhupesh Khanna

avatar
New Contributor

Hello,I met the same question as yours, but I really dont understand what the links say in your posts, could I get your explaination? just add the value of atlas.rest.address can solve this problem? But I have already add atlas.rest.address = <Atlas-host>:<port> property to atlas-application.properties,By the way, I am using atlas 1.2.0

Here is my atlas-application.properties:

atlas.graph.storage.backend=hbase
atlas.graph.storage.hbase.table=apache_atlas_janus


atlas.graph.storage.hostname=gaoenwei66.cmp3481.interconnect,gaoenwei67.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect
atlas.graph.storage.hbase.regions-per-server=1
atlas.graph.storage.lock.wait-time=10000

atlas.EntityAuditRepository.impl=org.apache.atlas.repository.audit.HBaseBasedAuditRepository

atlas.graph.index.search.backend=elasticsearch
atlas.graph.index.search.index-name=es_janusgraph

atlas.graph.index.search.hostname=gaoenwei70.cmp3481.interconnect,gaoenwei68.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect
atlas.graph.index.search.elasticsearch.client-only=true

atlas.graph.index.search.max-result-set-size=150

atlas.notification.embedded=false
#atlas.kafka.data=${sys:atlas.home}/data/kafka
atlas.kafka.zookeeper.connect=gaoenwei67.cmp3481.interconnect:2181,gaoenwei66.cmp3481.interconnect:2181,gaoenwei65.cmp3481.interconnect:2181
atlas.kafka.bootstrap.servers=gaoenwei67.cmp3481.interconnect:2181,gaoenwei66.cmp3481.interconnect:2181,gaoenwei65.cmp3481.interconnect:9092
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.connection.timeout.ms=200
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.auto.commit.interval.ms=1000
atlas.kafka.hook.group.id=atlas

atlas.kafka.enable.auto.commit=false
atlas.kafka.auto.offset.reset=earliest
atlas.kafka.session.timeout.ms=30000
atlas.kafka.offsets.topic.replication.factor=2
atlas.kafka.poll.timeout.ms=1000

atlas.notification.hook.topic.name=ATLAS_HOOK
atlas.notification.entities.topic.name=ATLAS_ENTITIES
 
atlas.notification.hook.consumer.topic.names=ATLAS_HOOK
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES

atlas.notification.create.topics=false
atlas.notification.replicas=2
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.notification.log.failed.messages=true
atlas.notification.consumer.retry.interval=500
atlas.notification.hook.retry.interval=1000

atlas.enableTLS=false

atlas.authentication.method.kerberos=false
atlas.authentication.method.file=true

#### ldap.type= LDAP or AD
atlas.authentication.method.ldap.type=none

#### user credentials file
atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties

atlas.rest.address=http://172.21.42.78:21000/
# If enabled and set to true, this will run setup steps when the server starts
atlas.server.run.setup.on.start=false

atlas.audit.hbase.tablename=apache_atlas_entity_audit
atlas.audit.zookeeper.session.timeout.ms=1000
atlas.audit.hbase.zookeeper.quorum=gaoenwei66.cmp3481.interconnect,gaoenwei67.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect:2181

atlas.server.ha.enabled=false

atlas.authorizer.impl=simple
atlas.authorizer.simple.authz.policy.file=atlas-simple-authz-policy.json

atlas.rest-csrf.enabled=true
atlas.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*,^Chrome.*
atlas.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE
atlas.rest-csrf.custom-header=X-XSRF-HEADER

atlas.metric.query.cache.ttlInSecs=900
atlas.search.gremlin.enable=false

atlas.hook.hive.synchronous=false
atlas.hook.hive.numRetries=3
atlas.hook.hive.queueSize=10000
atlas.cluster.name=primary
 
atlas.hook.hbase.synchronous=false
atlas.hook.hbase.numRetries=3 
atlas.hook.hbase.queueSize=10000 
atlas.cluster.name=primary

 

I really need your help, wish for your answer 

avatar
@Bhupesh Khanna

Please provide more details about your cluster. Is your cluster kerberized? Is ranger enabled? which version of HDP you have installed?

avatar
Rising Star

Hi @Ayub Khan

Below are the answers

Is your cluster kerberized? - No

Is ranger enabled? - Yes

which version of HDP you have installed? - HDP-2.5.0.0

please let me know if more information is required.

Thanks,

Bhupesh Khanna

avatar

Starting Kafka fixed this issue for me!