Member since
08-28-2017
26
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2795 | 03-06-2019 03:20 PM | |
6103 | 12-03-2018 05:52 AM |
12-06-2018
03:00 PM
Hello Team , We are using HDP-2.6.5.0 and Ambari-2.6.2.0. We have executed import-hive.sh script to import already existing hive tables in Atlas. It got successfully executed. Now we can see all hive databases and tables in Atlas, but we are not able to see data linage of that imported tables. If we create external table on any hdfs path then we can see lineage in Atlas. Also, If we create any managed tables in Atlas, we are not able to see lineage of that tables. Why we are not getting lineage of older tables and new managed tables? Please suggest. We are stuck now. Thanks, Owez
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hive
12-05-2018
06:32 AM
@Arshadullah Khan Thanks for the suggestion issue got resolved. Downloaded and Copied jar to /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl and issue got resolved after running import-hive.sh script. Now we are able to see Hive_db and Hive_tables in Atlas Web UI. Thanks & regards, Owez
... View more
12-04-2018
02:03 PM
@Sarath Subramanian yes i have copied some of the jars which were not there in below path, but the list you share that jars are already present in path. /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl -rw-r--r-- 1 atlas hadoop 4573750 May 11 2018 scala-reflect-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scalap-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 802818 May 11 2018 scalap-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scala-library-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 5744974 May 11 2018 scala-library-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scala-compiler-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 15487351 May 11 2018 scala-compiler-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt kafka-clients-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 1590487 May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt kafka_2.11-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 7532355 May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-native_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 67829 May 11 2018 json4s-native_2.11-3.2.11.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-core_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 589462 May 11 2018 json4s-core_2.11-3.2.11.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-ast_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 82421 May 11 2018 json4s-ast_2.11-3.2.11.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt jersey-multipart-1.19.jar
-rw-r--r-- 1 atlas hadoop 53275 May 11 2018 jersey-multipart-1.19.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hive-bridge-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 70540 May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hdfs-model-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 11639 May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hbase-common-1.1.2.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 577458 May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt gson-2.5.jar
-rw-r--r-- 1 atlas hadoop 232482 May 11 2018 gson-2.5.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-typesystem-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 408745 May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-notification-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 98220 May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-intg-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 222091 May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 64777 May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-v2-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 18039 May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-v1-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 43413 May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 35271 May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar
in application.log below is the error 2018-12-04 03:54:47,186 ERROR - [pool-2-thread-9 - 138daf6f-4389-4309-b63f-5d321670891e:] ~ graph rollback due to exception AtlasBaseException:Instance hive_db with unique attribute {qualifiedName=aem_develop@C03} does not exist (GraphTransactionInterceptor:73) could you please let me know what i have to do next to solve this issue. Thanks&Regards, Owez
... View more
12-03-2018
01:51 PM
@Sarath Subramanian Below are the jars present in /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrth
total 123M
-rw-r--r-- 1 atlas hadoop 4.4M May 11 2018 scala-reflect-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 785K May 11 2018 scalap-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 5.5M May 11 2018 scala-library-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 15M May 11 2018 scala-compiler-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 1.6M May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 7.2M May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 67K May 11 2018 json4s-native_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 576K May 11 2018 json4s-core_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 81K May 11 2018 json4s-ast_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 53K May 11 2018 jersey-multipart-1.19.jar
-rw-r--r-- 1 atlas hadoop 69K May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 12K May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 564K May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 228K May 11 2018 gson-2.5.jar
-rw-r--r-- 1 atlas hadoop 400K May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 96K May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 217K May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 64K May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 18K May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 43K May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 35K May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 root root 141K Nov 29 11:19 commons-digester.jar
-rw-r--r-- 1 root root 141K Nov 29 11:19 commons-digester-1.8.jar
-rw-r--r-- 1 atlas hadoop 292K Nov 29 11:19 commons-configuration.jar
-rw-r--r-- 1 atlas hadoop 292K Nov 29 11:19 commons-configuration-1.6.jar
-rw-r--r-- 1 root root 236K Nov 29 11:19 commons-compress.jar
-rw-r--r-- 1 root root 236K Nov 29 11:19 commons-compress-1.4.1.jar
-rw-r--r-- 1 root root 575K Nov 29 11:19 commons-collections.jar
-rw-r--r-- 1 root root 575K Nov 29 11:19 commons-collections-3.2.2.jar
-rw-r--r-- 1 root root 57K Nov 29 11:19 commons-codec.jar
-rw-r--r-- 1 root root 57K Nov 29 11:19 commons-codec-1.4.jar
-rw-r--r-- 1 root root 41K Nov 29 11:19 commons-cli.jar
-rw-r--r-- 1 root root 41K Nov 29 11:19 commons-cli-1.2.jar
-rw-r--r-- 1 root root 185K Nov 29 11:19 commons-beanutils.jar
-rw-r--r-- 1 root root 202K Nov 29 11:19 commons-beanutils-core.jar
-rw-r--r-- 1 root root 202K Nov 29 11:19 commons-beanutils-core-1.8.0.jar
-rw-r--r-- 1 root root 185K Nov 29 11:19 commons-beanutils-1.7.0.jar
-rw-r--r-- 1 root root 768K Nov 29 11:19 azure-storage.jar
-rw-r--r-- 1 root root 768K Nov 29 11:19 azure-storage-5.4.0.jar
-rw-r--r-- 1 root root 9.9K Nov 29 11:19 azure-keyvault-core.jar
-rw-r--r-- 1 root root 9.9K Nov 29 11:19 azure-keyvault-core-0.8.0.jar
-rw-r--r-- 1 root root 297K Nov 29 11:19 avro.jar
-rw-r--r-- 1 root root 297K Nov 29 11:19 avro-1.7.4.jar
-rw-r--r-- 1 root root 79K Nov 29 11:19 api-util.jar
-rw-r--r-- 1 root root 79K Nov 29 11:19 api-util-1.0.0-M20.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 api-asn1-api.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 api-asn1-api-1.0.0-M20.jar
-rw-r--r-- 1 root root 676K Nov 29 11:19 apacheds-kerberos-codec.jar
-rw-r--r-- 1 root root 676K Nov 29 11:19 apacheds-kerberos-codec-2.0.0-M15.jar
-rw-r--r-- 1 root root 44K Nov 29 11:19 apacheds-i18n.jar
-rw-r--r-- 1 root root 44K Nov 29 11:19 apacheds-i18n-2.0.0-M15.jar
-rw-r--r-- 1 root root 62K Nov 29 11:19 activation.jar
-rw-r--r-- 1 root root 62K Nov 29 11:19 activation-1.1.jar
-rw-r--r-- 1 root root 40K Nov 29 11:19 hadoop-mapreduce-client-jobclient.jar
-rw-r--r-- 1 root root 40K Nov 29 11:19 hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 hadoop-mapreduce-client-core.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 737K Nov 29 11:19 hadoop-mapreduce-client-common.jar
-rw-r--r-- 1 root root 737K Nov 29 11:19 hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 509K Nov 29 11:19 hadoop-mapreduce-client-app.jar
-rw-r--r-- 1 root root 509K Nov 29 11:19 hadoop-mapreduce-client-app-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 8.4M Nov 29 11:19 hadoop-hdfs.jar
-rw-r--r-- 1 root root 8.4M Nov 29 11:19 hadoop-hdfs-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 3.6M Nov 29 11:19 hadoop-common.jar
-rw-r--r-- 1 root root 3.6M Nov 29 11:19 hadoop-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 78K Nov 29 11:19 hadoop-auth.jar
-rw-r--r-- 1 root root 78K Nov 29 11:19 hadoop-auth-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 hadoop-annotations.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 hadoop-annotations-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 guava.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 guava-11.0.2.jar
-rw-r--r-- 1 root root 186K Nov 29 11:19 gson.jar
-rw-r--r-- 1 root root 186K Nov 29 11:19 gson-2.2.4.jar
-rw-r--r-- 1 root root 265K Nov 29 11:19 curator-recipes.jar
-rw-r--r-- 1 root root 265K Nov 29 11:19 curator-recipes-2.7.1.jar
-rw-r--r-- 1 root root 182K Nov 29 11:19 curator-framework.jar
-rw-r--r-- 1 root root 182K Nov 29 11:19 curator-framework-2.7.1.jar
-rw-r--r-- 1 root root 68K Nov 29 11:19 curator-client.jar
-rw-r--r-- 1 root root 68K Nov 29 11:19 curator-client-2.7.1.jar
-rw-r--r-- 1 root root 267K Nov 29 11:19 commons-net.jar
-rw-r--r-- 1 root root 267K Nov 29 11:19 commons-net-3.1.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 commons-math3.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 commons-math3-3.1.1.jar
-rw-r--r-- 1 root root 61K Nov 29 11:19 commons-logging.jar
-rw-r--r-- 1 root root 61K Nov 29 11:19 commons-logging-1.1.3.jar
-rw-r--r-- 1 root root 278K Nov 29 11:19 commons-lang.jar
-rw-r--r-- 1 root root 425K Nov 29 11:19 commons-lang3.jar
-rw-r--r-- 1 root root 425K Nov 29 11:19 commons-lang3-3.4.jar
-rw-r--r-- 1 root root 278K Nov 29 11:19 commons-lang-2.6.jar
-rw-r--r-- 1 root root 181K Nov 29 11:19 commons-io.jar
-rw-r--r-- 1 root root 181K Nov 29 11:19 commons-io-2.4.jar
-rw-r--r-- 1 root root 27K Nov 29 11:19 jackson-xc.jar
-rw-r--r-- 1 root root 27K Nov 29 11:19 jackson-xc-1.9.13.jar
-rw-r--r-- 1 root root 763K Nov 29 11:19 jackson-mapper-asl.jar
-rw-r--r-- 1 root root 763K Nov 29 11:19 jackson-mapper-asl-1.9.13.jar
-rw-r--r-- 1 root root 18K Nov 29 11:19 jackson-jaxrs.jar
-rw-r--r-- 1 root root 18K Nov 29 11:19 jackson-jaxrs-1.9.13.jar
-rw-r--r-- 1 root root 846K Nov 29 11:19 jackson-databind.jar
-rw-r--r-- 1 root root 846K Nov 29 11:19 jackson-databind-2.2.3.jar
-rw-r--r-- 1 root root 189K Nov 29 11:19 jackson-core.jar
-rw-r--r-- 1 root root 227K Nov 29 11:19 jackson-core-asl.jar
-rw-r--r-- 1 root root 227K Nov 29 11:19 jackson-core-asl-1.9.13.jar
-rw-r--r-- 1 root root 189K Nov 29 11:19 jackson-core-2.2.3.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jackson-annotations.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jackson-annotations-2.2.3.jar
-rw-r--r-- 1 root root 320K Nov 29 11:19 httpcore.jar
-rw-r--r-- 1 root root 320K Nov 29 11:19 httpcore-4.4.4.jar
-rw-r--r-- 1 root root 720K Nov 29 11:19 httpclient.jar
-rw-r--r-- 1 root root 720K Nov 29 11:19 httpclient-4.5.2.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 htrace-core.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 htrace-core-3.1.0-incubating.jar
-rw-r--r-- 1 root root 414K Nov 29 11:19 hadoop-yarn-server-common.jar
-rw-r--r-- 1 root root 414K Nov 29 11:19 hadoop-yarn-server-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 100K Nov 29 11:19 hadoop-yarn-registry.jar
-rw-r--r-- 1 root root 100K Nov 29 11:19 hadoop-yarn-registry-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.8M Nov 29 11:19 hadoop-yarn-common.jar
-rw-r--r-- 1 root root 1.8M Nov 29 11:19 hadoop-yarn-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 198K Nov 29 11:19 hadoop-yarn-client.jar
-rw-r--r-- 1 root root 198K Nov 29 11:19 hadoop-yarn-client-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 2.1M Nov 29 11:19 hadoop-yarn-api.jar
-rw-r--r-- 1 root root 2.1M Nov 29 11:19 hadoop-yarn-api-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 54K Nov 29 11:19 hadoop-mapreduce-client-shuffle.jar
-rw-r--r-- 1 root root 54K Nov 29 11:19 hadoop-mapreduce-client-shuffle-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:19 netty.jar
-rw-r--r-- 1 root root 2.2M Nov 29 11:19 netty-all.jar
-rw-r--r-- 1 root root 2.2M Nov 29 11:19 netty-all-4.0.52.Final.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:19 netty-3.6.2.Final.jar
-rw-r--r-- 1 root root 479K Nov 29 11:19 log4j.jar
-rw-r--r-- 1 root root 479K Nov 29 11:19 log4j-1.2.17.jar
-rw-r--r-- 1 root root 1022K Nov 29 11:19 leveldbjni-all.jar
-rw-r--r-- 1 root root 1022K Nov 29 11:19 leveldbjni-all-1.8.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jsr305.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jsr305-3.0.0.jar
-rw-r--r-- 1 root root 99K Nov 29 11:19 jsp-api.jar
-rw-r--r-- 1 root root 99K Nov 29 11:19 jsp-api-2.1.jar
-rw-r--r-- 1 root root 77K Nov 29 11:19 json-smart.jar
-rw-r--r-- 1 root root 77K Nov 29 11:19 json-smart-1.3.1.jar
-rw-r--r-- 1 root root 174K Nov 29 11:19 jetty-util.jar
-rw-r--r-- 1 root root 174K Nov 29 11:19 jetty-util-6.1.26.hwx.jar
-rw-r--r-- 1 root root 19K Nov 29 11:19 jetty-sslengine.jar
-rw-r--r-- 1 root root 19K Nov 29 11:19 jetty-sslengine-6.1.26.hwx.jar
-rw-r--r-- 1 root root 448K Nov 29 11:19 jersey-core.jar
-rw-r--r-- 1 root root 448K Nov 29 11:19 jersey-core-1.9.jar
-rw-r--r-- 1 root root 128K Nov 29 11:19 jersey-client.jar
-rw-r--r-- 1 root root 128K Nov 29 11:19 jersey-client-1.9.jar
-rw-r--r-- 1 root root 4.7K Nov 29 11:19 jcip-annotations.jar
-rw-r--r-- 1 root root 4.7K Nov 29 11:19 jcip-annotations-1.0-1.jar
-rw-r--r-- 1 root root 103K Nov 29 11:19 jaxb-api.jar
-rw-r--r-- 1 root root 103K Nov 29 11:19 jaxb-api-2.2.2.jar
-rw-r--r-- 1 root root 15K Nov 29 11:20 xmlenc.jar
-rw-r--r-- 1 root root 15K Nov 29 11:20 xmlenc-0.52.jar
-rw-r--r-- 1 root root 190K Nov 29 11:20 xml-apis.jar
-rw-r--r-- 1 root root 190K Nov 29 11:20 xml-apis-1.3.04.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:20 xercesImpl.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:20 xercesImpl-2.9.1.jar
-rw-r--r-- 1 root root 23K Nov 29 11:20 stax-api.jar
-rw-r--r-- 1 root root 23K Nov 29 11:20 stax-api-1.0-2.jar
-rw-r--r-- 1 root root 973K Nov 29 11:20 snappy-java.jar
-rw-r--r-- 1 root root 973K Nov 29 11:20 snappy-java-1.0.4.1.jar
-rw-r--r-- 1 root root 8.7K Nov 29 11:20 slf4j-log4j12.jar
-rw-r--r-- 1 root root 8.7K Nov 29 11:20 slf4j-log4j12-1.7.10.jar
-rw-r--r-- 1 root root 32K Nov 29 11:20 slf4j-api.jar
-rw-r--r-- 1 root root 32K Nov 29 11:20 slf4j-api-1.7.10.jar
-rw-r--r-- 1 root root 103K Nov 29 11:20 servlet-api.jar
-rw-r--r-- 1 root root 103K Nov 29 11:20 servlet-api-2.5.jar
-rw-r--r-- 1 root root 521K Nov 29 11:20 protobuf-java.jar
-rw-r--r-- 1 root root 521K Nov 29 11:20 protobuf-java-2.5.0.jar
-rw-r--r-- 1 root root 29K Nov 29 11:20 paranamer.jar
-rw-r--r-- 1 root root 29K Nov 29 11:20 paranamer-2.3.jar
-rw-r--r-- 1 root root 65K Nov 29 11:20 okio.jar
-rw-r--r-- 1 root root 65K Nov 29 11:20 okio-1.6.0.jar
-rw-r--r-- 1 root root 324K Nov 29 11:20 okhttp.jar
-rw-r--r-- 1 root root 324K Nov 29 11:20 okhttp-2.7.5.jar
-rw-r--r-- 1 root root 293K Nov 29 11:20 nimbus-jose-jwt.jar
-rw-r--r-- 1 root root 293K Nov 29 11:20 nimbus-jose-jwt-4.41.1.jar
-rw-r--r-- 1 root root 785K Nov 29 11:20 zookeeper.jar
-rw-r--r-- 1 root root 785K Nov 29 11:20 zookeeper-3.4.6.2.6.5.0-292.jar
-rw-r--r-- 1 root root 93K Nov 29 11:20 xz.jar
-rw-r--r-- 1 root root 93K Nov 29 11:20 xz-1.0.jar
root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl#
root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# pwd
/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl
root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt |wc -l
180 Could you please let me know which jars are missing or any sym links are missing. Below is the current issue while running import-hive.sh script. -rwxr-xr-x 1 atlas hadoop 3652 May 11 2018 import-hive.sh*
root@hadwkrndcc03-3:/usr/hdp/2.6.5.0-292/atlas/hook-bin# ./import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/atlas/hook/hive/atlas-hive-plugin-impl/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/atlas/hook/hive/atlas-hive-plugin-impl/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hive/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout.
Enter username for atlas :- admin
Enter password for atlas :-
Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jettison/json/JSONException
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:124)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jettison.json.JSONException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
Failed to import Hive Meta Data!!!
root@hadwkrndcc03-3:/usr/hdp/2.6.5.0-292/atlas/hook-bin# Thanks&Regards, Owez Mujawar
... View more
12-03-2018
05:52 AM
@Sarath Subramanian now i am able publish a sample message to ATLAS_ENTITIES. Resolution: Deleted the older Atlas_entities and created a new one. Thanks for the help. Regards, Owez Mujawar
... View more
11-29-2018
03:06 PM
@Sarath Subramanian Please find below issue while publishing message to ATLAS_ENTITIES. atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-producer.sh --broker-list hadwkrndcc03-3.lifeway.org:6667 --topic ATLAS_ENTITIES
>olla
>[2018-11-29 05:50:15,694] ERROR Error when sending message to topic ATLAS_ENTITIES with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 1526 ms has passed since batch creation plus linger time Could you please help in this issue and provide a solution for it. Thanks in advance. Regards, Owez Mujawar
... View more
11-29-2018
02:20 PM
@Sarath Subramanian Hi, As we are trying to execute the query but we are failing. below error for the same. could you please help me to resolved this issue. atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/current/atlas-server/bin/quick_start.py
Enter username for atlas :- admin
Enter password for atlas :-
Creating sample types:
Created type [DB]
Created type [Table]
Created type [StorageDesc]
Created type [Column]
Created type [LoadProcess]
Created type [View]
Created type [JdbcAccess]
Created type [ETL]
Created type [Metric]
Created type [PII]
Created type [Fact]
Created type [Dimension]
Created type [Log Data]
Creating sample entities:
Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClientV2$API_V2@514646ef failed with status 500 (Internal Server Error) Response Body ({"errorCode":"ATLAS-500-00-007","errorMessage":"Failed to notify NotificationEntityChangeListener for change CREATE"})
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:387)
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:320)
at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:208)
at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:281)
at org.apache.atlas.examples.QuickStartV2.createInstance(QuickStartV2.java:339)
at org.apache.atlas.examples.QuickStartV2.createDatabase(QuickStartV2.java:362)
at org.apache.atlas.examples.QuickStartV2.createEntities(QuickStartV2.java:268)
at org.apache.atlas.examples.QuickStartV2.runQuickstart(QuickStartV2.java:150)
at org.apache.atlas.examples.QuickStartV2.main(QuickStartV2.java:132)
No sample data added to Apache Atlas Server. Thanks in advance. Regards, Owez Mujawar
... View more
11-29-2018
02:19 PM
@Sarath Subramanian As we have copied commons-configuration.jar is given path. /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl then Again we have tried to run the script but still issue persist. /usr/hdp/2.6.5.0-292/atlas/hook-bin# ./import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout.
Enter username for atlas :- admin
Enter password for atlas :-
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/ClientHandlerException
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:124)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.ClientHandlerException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
Failed to import Hive Meta Data!!! Could you please help me with this. Thanks in Advance. Regards, Owez Mujawar
... View more
11-28-2018
03:06 PM
Hi team, As i am trying to run import-hive.sh script but it is not able to excute it. below are the logs attached. /usr/hdp/2.6.5.0-292/atlas/hook-bin/import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/PropertiesConfiguration
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:112)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.PropertiesConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
Failed to import Hive Meta Data!!! Could you please help me with this. Thanks in Advance. Regards, Owez Mujawar
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hadoop
-
Apache Hive
11-28-2018
03:02 PM
Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0.
While creating new entity in Atlas we are getting following error in application.log. Currently we are not using Ranger. we trying to create new entity without ranger. Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:220)
at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:182)
at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:117)
at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:208)
at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:102)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityC
hangeNotifier.java:176)
... 100 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:207)
... 105 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ ==> releaseLockedObjects(): lockedGuids.size: 0 (GraphTransactionInterceptor:187)
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ <== releaseLockedObjects(): lockedGuids.size: 0 (GraphTransactionInterceptor:197)
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ PERF|EntityREST.createOrUpdate()|30228 (EntityREST:77)
2018-11-28 01:35:02,638 ERROR - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ Error handling a request: 168f59a001543090 (ExceptionMapperUtil:32)
org.apache.atlas.exception.AtlasBaseException: Failed to notify NotificationEntityChangeListener for change UPDATE
at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:183)
I can describe topic for ATLAS_ENTITIES /usr/hdp/2.6.5.0-292/kafka/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper hadmgrndcc03-1.lifeway.org:2181,hadmgrndcc03-2.lifeway.org:2181,hadmgrndcc03-3.lifeway.org:2181
Topic:ATLAS_ENTITIES PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ATLAS_ENTITIES Partition: 0 Leader: 1003 Replicas: 1003 Isr: 1003
ACL Permission:-
/usr/hdp/2.6.5.0-292/kafka/bin/kafka-acls.sh --add --allow-principal user:* --producer --topic ATLAS_ENTITIES --authorizer-properties zookeeper.connect=hadmgrndcc03-1.lifeway.org:2181,hadmgrndcc03-2.lifeway.org:2181,hadmgrndcc03-3.lifeway.org:2181
Adding ACLs for resource `Topic:ATLAS_ENTITIES`:
user:* has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Describe from hosts: *
Adding ACLs for resource `Cluster:kafka-cluster`:
user:* has Allow permission for operations: Create from hosts: *
Current ACLs for resource `Topic:ATLAS_ENTITIES`:
User: has Allow permission for operations: Read from hosts: *
User: has Allow permission for operations: Describe from hosts: *
user:* has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Write from hosts: application.log
User:* has Allow permission for operations: Describe from hosts: *
User:atlas has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Read from hosts: application.log
User:* has Allow permission for operations: Read from hosts: *
User:atlas has Allow permission for operations: Describe from hosts: * Could you please help in this issue and provide a solution for it. Thanks in advance. Regards,
Owez Mujawar
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Kafka
- « Previous
-
- 1
- 2
- Next »