question Re: Apache atlas [import-hive.sh failed] in Archives of Support Questions (Read Only)

Apache atlas [import-hive.sh failed]

bkhanna — Thu, 20 Oct 2016 21:06:19 GMT

Atlas error while running import-hive.sh

Anycan help me..complete stack trace as below.

2016-10-20 19:23:36,626 DEBUG - [main:] ~ Using resource http://ServerName:21000/api/atlas/entities/4dc94048-3f95-42f7-ae5f-6e9b5baea06d for 0 times (AtlasClient:784) 2016-10-20 19:24:36,673 WARN - [main:] ~ Handled exception in calling api api/atlas/entities (AtlasClient:791) com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623) at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188) at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785) at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 14 more 2016-10-20 19:24:36,675 WARN - [main:] ~ Exception's cause: class java.net.SocketTimeoutException (AtlasClient:792) Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81) at com.sun.jersey.api.client.Client.handle(Client.java:648) at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670) at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623) at org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:1188) at org.apache.atlas.AtlasClient.callAPIWithRetries(AtlasClient.java:785) at org.apache.atlas.AtlasClient.callAPI(AtlasClient.java:1214) at org.apache.atlas.AtlasClient.updateEntity(AtlasClient.java:808) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.updateInstance(HiveMetaStoreBridge.java:506) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:159) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:124) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:118) at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:662) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240) at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147) ... 14 more Failed to import Hive Data Model!!!

Re: Apache atlas [import-hive.sh failed]

bkhanna — Thu, 20 Oct 2016 21:42:28 GMT

@Ayub Khan

I have seen you have resolved similar kind of problem with Atlas, could you please help me on the same

i used some references

http://atlas.incubator.apache.org/0.7.0-incubating/Bridge-Hive.html

https://community.hortonworks.com/questions/39839/how-to-import-metadata-from-hive-into-atlas-and-th.html

Thanks in advance

Bhupesh Khanna

Re: Apache atlas [import-hive.sh failed]

apathan — Thu, 20 Oct 2016 22:39:46 GMT

@Bhupesh Khanna

Please provide more details about your cluster. Is your cluster kerberized? Is ranger enabled? which version of HDP you have installed?

Re: Apache atlas [import-hive.sh failed]

bkhanna — Fri, 21 Oct 2016 12:03:39 GMT

Hi @Ayub Khan

Below are the answers

Is your cluster kerberized? - No

Is ranger enabled? - Yes

which version of HDP you have installed? - HDP-2.5.0.0

please let me know if more information is required.

Thanks,

Bhupesh Khanna

Re: Apache atlas [import-hive.sh failed]

jannik_franz — Thu, 10 Nov 2016 16:20:03 GMT

Starting Kafka fixed this issue for me!

Re: Apache atlas [import-hive.sh failed]

Minyan — Wed, 10 Nov 2021 02:37:09 GMT

Hello，I met the same question as yours, but I really dont understand what the links say in your posts, could I get your explaination? just add the value of atlas.rest.address can solve this problem? But I have already add atlas.rest.address = <Atlas-host>:<port> property to atlas-application.properties，By the way, I am using atlas 1.2.0

Here is my atlas-application.properties:

atlas.graph.storage.backend=hbase atlas.graph.storage.hbase.table=apache_atlas_janus atlas.graph.storage.hostname=gaoenwei66.cmp3481.interconnect,gaoenwei67.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect atlas.graph.storage.hbase.regions-per-server=1 atlas.graph.storage.lock.wait-time=10000 atlas.EntityAuditRepository.impl=org.apache.atlas.repository.audit.HBaseBasedAuditRepository atlas.graph.index.search.backend=elasticsearch atlas.graph.index.search.index-name=es_janusgraph atlas.graph.index.search.hostname=gaoenwei70.cmp3481.interconnect,gaoenwei68.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect atlas.graph.index.search.elasticsearch.client-only=true atlas.graph.index.search.max-result-set-size=150 atlas.notification.embedded=false #atlas.kafka.data=${sys:atlas.home}/data/kafka atlas.kafka.zookeeper.connect=gaoenwei67.cmp3481.interconnect:2181,gaoenwei66.cmp3481.interconnect:2181,gaoenwei65.cmp3481.interconnect:2181 atlas.kafka.bootstrap.servers=gaoenwei67.cmp3481.interconnect:2181,gaoenwei66.cmp3481.interconnect:2181,gaoenwei65.cmp3481.interconnect:9092 atlas.kafka.zookeeper.session.timeout.ms=400 atlas.kafka.zookeeper.connection.timeout.ms=200 atlas.kafka.zookeeper.sync.time.ms=20 atlas.kafka.auto.commit.interval.ms=1000 atlas.kafka.hook.group.id=atlas atlas.kafka.enable.auto.commit=false atlas.kafka.auto.offset.reset=earliest atlas.kafka.session.timeout.ms=30000 atlas.kafka.offsets.topic.replication.factor=2 atlas.kafka.poll.timeout.ms=1000 atlas.notification.hook.topic.name=ATLAS_HOOK atlas.notification.entities.topic.name=ATLAS_ENTITIES atlas.notification.hook.consumer.topic.names=ATLAS_HOOK atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES atlas.notification.create.topics=false atlas.notification.replicas=2 atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES atlas.notification.log.failed.messages=true atlas.notification.consumer.retry.interval=500 atlas.notification.hook.retry.interval=1000 atlas.enableTLS=false atlas.authentication.method.kerberos=false atlas.authentication.method.file=true #### ldap.type= LDAP or AD atlas.authentication.method.ldap.type=none #### user credentials file atlas.authentication.method.file.filename=${sys:atlas.home}/conf/users-credentials.properties atlas.rest.address=http://172.21.42.78:21000/ # If enabled and set to true, this will run setup steps when the server starts atlas.server.run.setup.on.start=false atlas.audit.hbase.tablename=apache_atlas_entity_audit atlas.audit.zookeeper.session.timeout.ms=1000 atlas.audit.hbase.zookeeper.quorum=gaoenwei66.cmp3481.interconnect,gaoenwei67.cmp3481.interconnect,gaoenwei65.cmp3481.interconnect:2181 atlas.server.ha.enabled=false atlas.authorizer.impl=simple atlas.authorizer.simple.authz.policy.file=atlas-simple-authz-policy.json atlas.rest-csrf.enabled=true atlas.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*,^Chrome.* atlas.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE atlas.rest-csrf.custom-header=X-XSRF-HEADER atlas.metric.query.cache.ttlInSecs=900 atlas.search.gremlin.enable=false atlas.hook.hive.synchronous=false atlas.hook.hive.numRetries=3 atlas.hook.hive.queueSize=10000 atlas.cluster.name=primary atlas.hook.hbase.synchronous=false atlas.hook.hbase.numRetries=3 atlas.hook.hbase.queueSize=10000 atlas.cluster.name=primary

I really need your help, wish for your answer