Hi @Anpan It is better to have one AD for each environment. You can control each user with setting up a different principal's like hdfs principal for DEV users and separate for PROD users. You can easy maintained different OU and Groups for each Environment as per requirement in your organisation. It is also recommended to have only one AD server for each environment. This will be cost effective and easy to manage OU , Groups and Princiapls etc.. Thanks, Owez ... View more

Hi Balaji, Could you please switch your user to hdfs with "hdpuser" then try to put the files on hdfs. And also could you please check your hdfs policy in Ranger. Thanks, Owez ... View more

@Ahmad Hajitorab , Try to run sample metadata in Atlas. su atlas -c '/usr/hdp/current/atlas-server/bin/quick_start.py' after then check application.log. Questions:- Have you enabled Plugin's and Have you set Audit to Solr? You can also run import-hive.sh. Please share your logs details. ... View more

@Vinay Could you please check your ambari-agent and ambari-server version. check in database as well. This issue occurs when there are incorrect entries present in the ambari database. The following is the table in the database. From the host_version ambari=>select*from host_version; Thanks, Owez ... View more

@Kishore Jannu Could you try below solution. it has worked for me. mv /var/lib/ambari-agent/data/structured-out-status.json /var/lib/ambari-agent/data/structured-out-status.json-old Try to restart ambari-server restart ambari-agent restart Or Could you please check your value for dfs.permissions.superusergroup. This might be causing your issue. Try to modify the dfs.permissions.superusergroup variable from hdfs. Could you please let me know if issue is still persist. Thanks, Owez ... View more

@Sharmadha Sainath Please find attachment for your reference. We have offsets.topic.replication.factor=1 and we have 1 Kafka broker. We haven't found any error in Kafka logs. KAFKA Logs: [2018-12-13 04:29:43,286] INFO Updated PartitionLeaderEpoch. New: {epoch:16, offset:1063289}, Current: {epoch:14, offset916036} for Partition: ATLAS_ENTITIES-0. Cache now contains 8 entries. (kafka.server.epoch.LeaderEpochFileCache) [2018-12-13 04:34:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) [2018-12-13 04:44:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) [2018-12-13 04:54:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) Still we haven't fount any solution. Could you please help me to resolved this issue on priority. Thanks, Owez ... View more

@Sharmadha Sainath Hive Hook is enabled Please find attachment for your reference. Checked hive permission and Atlas that is perfect no issues in Permission. Still when we are creating new tables it is not reflecting in Atlas ui. we need to run import-hive.sh. We are getting below error when we run import-hive.sh script. it is coming in application.log in Atlas rest services has no errors in their logs. This errors are for newly created table. but still due to this error also we are able to see external table and its lineage. 2018-12-13 04:32:58,136 ERROR - [pool-2-thread-9 - 04e4ba41-c09d-4257-9391-1fde800fa5d2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep13@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:32:59,132 ERROR - [pool-2-thread-10 - fd8d6366-e465-450a-a42a-8d98022787f2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep2@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:01,145 ERROR - [pool-2-thread-9 - bceb4f09-1cce-461d-8b1a-2e4a90e39717:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.dep2@C03:1544606069000} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:04,374 ERROR - [pool-2-thread-7 - e62c07bc-4fe4-480a-bf80-6ffff789c698:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee19@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:04,975 ERROR - [pool-2-thread-8 - d2f31e6c-fe99-40d8-a153-087b9c10eb8b:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee20@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:05,347 ERROR - [pool-2-thread-9 - 4f478d9e-4c32-456e-8669-0d4bcef9bcf8:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data0@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:06,139 ERROR - [pool-2-thread-9 - e6da522f-ff85-436b-b66b-2844ca2fdf2c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data04@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:06,388 ERROR - [pool-2-thread-8 - a385f421-d818-4fc6-a613-880064babcf9:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data04@C03:1544688356000} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:07,468 ERROR - [pool-2-thread-10 - e5c8525c-7877-4e1a-b2e6-b23d9f054e94:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data18@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:07,959 ERROR - [pool-2-thread-9 - 72143f28-2630-4306-8639-7c513dffc51c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data2@C03:1544605910000} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:08,180 ERROR - [pool-2-thread-10 - 516cb986-69ae-48f9-813f-3e7b23a1b967:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data20@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-13 04:33:08,439 ERROR - [pool-2-thread-9 - 5706eb89-2951-4c33-b914-9ace101cb011:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data3@C03} does not exist (GraphTransactionInterceptor:73) After running import script we are able to see only External tables in Atlas. no lineage for Managed and View tables. Could you please help me on this. Thanks, Owez ... View more

@Sharmadha Sainath Thanks for the confirmation. The scenario you explain that's perfect suits in our environment, but 3rd Scenario is not working in our environment. for example we are using below query to create Managed,View and External table: CREATE TABLE IF NOT EXISTS employee13 ( eid int, name int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/hive/data/employee13'; CREATE VIEW employee_data AS SELECT * FROM employee WHERE eid >0; CREATE EXTERNAL TABLE employee_data2 ( eid int, name int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOCATION '/hive/data/employee'; When we execute above query its gets successfully executed. we don't see any errors in logs. When we login to Atlas UI we are not able to see that newly created tables in 1st Attempt. After running import-hive.sh script we are able to see the lineage of External tables only. We are not able see lineage of newly created managed and view tables.it comes as no linage found. Could you please guide me on that. ... View more

@Sharmadha Sainath Hive Hook is enabled and we can see all tables and linage of External table only. We want to see lineage of Managed and Virtual view tables which were created before atlas. Could you please help me to see that lineage of Managed and Virtual view tables. i have not seen any error in ranger-log , kafka-log , and hive-log. only in application-log i can see below error. 2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73) 2018-12-11 00:11:19,247 INFO - [main-SendThread(hadmgrndcc03-1.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-1.lifeway.org/172.17.20.29:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019) 2018-12-11 00:11:30,951 INFO - [main-SendThread(hadmgrndcc03-3.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-3.lifeway.org/172.17.20.33:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019) 2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73) Let me know how we can see lineage of Managed and virtual view tables. ... View more

@Ronak bansal We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 We are getting below error while running import-hive.sh script. root@hn:/var/log/atlas# tail -f application.log 2018-12-10 04:50:06,783 ERROR - [pool-2-thread-7 - d35c5859-a93e-40ce-94ff-5c12e45f2880:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_awood@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:07,100 ERROR - [pool-2-thread-7 - 0b130511-ed2a-4434-95ad-692b0babf348:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:07,431 ERROR - [pool-2-thread-7 - fbf49dd4-12fd-4ff1-bdf7-d69187ead26c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2_new@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:07,826 ERROR - [pool-2-thread-7 - 1041077f-a6cf-4b24-8669-d84d88155d82:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:08,056 ERROR - [pool-2-thread-9 - ce6d8709-db1a-458e-8cfb-4d1ccbb28ca6:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_cdc@C03} does not exist (GraphTransactionInterceptor:73) 2018-12-10 04:50:08,332 ERROR - [pool-2-thread-7 - 1757d3a0-4562-4c9b-bf1b-526cd34b2c05:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_current@C03} does not exist (GraphTransactionInterceptor:73) In Atlas we are able to see External tables linage but not for Managed and Virtual Views tables. Could you please let me why we are getting this error. Can we see linage for Managed and Virtual view tables? Thanks, Owez ... View more

@Sharmadha Sainath @Ana Gillan Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Questions: 1) We need to know can we see linage of Managed tables and virtual views linage in Atlas? 2) When we create a new tables and try to see linage of that tables in first attempt we are not able to see linage of that tables after running import-hive.sh script we are able to see the linage of that newly created tables, why this is happening? it should show automatically the linage of newly created tables. 3) Could you please let me know is there any property need to set or any configuration needs to change. Two snap shot has been attached 1) atlas_external table as we can see linage of that table after running import-hive.sh script. 2) atlas_managed table we cannot see linage of that table even after running script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables and virtual views as well? There is no error in application.log, hbase-log, ranger-audit, kafka-log. Permission has been set in ranger. Please suggest. We are stuck now. Thanks, Owez ... View more

@artmro Could you please check the logs for Atlas, hbase and ranger-audit. You will find the error. please post that error. i have resolved that issue. Actually my issue was for jersey-jar issue. Let us know the error logs so that we can assist you on the same. Thanks, Owez ... View more

Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Now we are able to see linage of older tables as well, but only for External tables. As when we are trying to create a new managed tables and when we create a joins/intersecting two tables, we are not able to see their linage. if we create a new tables as external tables whereas we can see the linage of that tables after running import-hive.sh script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables as well? @team Please let me know what we can do in Atlas and what we cannot do in Atlas. ... View more

@Arshadullah Khan Even i have faced same issue Actually it doesn't work. i think this is a bug. As we should connect to at least one hive server when HA is configured. Bug is in Knox Gateway. ... View more

@Arshadullah Khan Thanks for the suggestion issue got resolved. Downloaded and Copied jar to /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl and issue got resolved after running import-hive.sh script. Now we are able to see Hive_db and Hive_tables in Atlas Web UI. Thanks & regards, Owez ... View more

@Sarath Subramanian yes i have copied some of the jars which were not there in below path, but the list you share that jars are already present in path. /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl -rw-r--r-- 1 atlas hadoop 4573750 May 11 2018 scala-reflect-2.11.8.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scalap-2.11.8.jar -rw-r--r-- 1 atlas hadoop 802818 May 11 2018 scalap-2.11.8.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scala-library-2.11.8.jar -rw-r--r-- 1 atlas hadoop 5744974 May 11 2018 scala-library-2.11.8.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scala-compiler-2.11.8.jar -rw-r--r-- 1 atlas hadoop 15487351 May 11 2018 scala-compiler-2.11.8.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt kafka-clients-1.0.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 1590487 May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt kafka_2.11-1.0.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 7532355 May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-native_2.11-3.2.11.jar -rw-r--r-- 1 atlas hadoop 67829 May 11 2018 json4s-native_2.11-3.2.11.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-core_2.11-3.2.11.jar -rw-r--r-- 1 atlas hadoop 589462 May 11 2018 json4s-core_2.11-3.2.11.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-ast_2.11-3.2.11.jar -rw-r--r-- 1 atlas hadoop 82421 May 11 2018 json4s-ast_2.11-3.2.11.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt jersey-multipart-1.19.jar -rw-r--r-- 1 atlas hadoop 53275 May 11 2018 jersey-multipart-1.19.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hive-bridge-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 70540 May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hdfs-model-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 11639 May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hbase-common-1.1.2.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 577458 May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt gson-2.5.jar -rw-r--r-- 1 atlas hadoop 232482 May 11 2018 gson-2.5.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-typesystem-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 408745 May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-notification-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 98220 May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-intg-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 222091 May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-common-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 64777 May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-v2-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 18039 May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-v1-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 43413 May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-common-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 35271 May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar in application.log below is the error 2018-12-04 03:54:47,186 ERROR - [pool-2-thread-9 - 138daf6f-4389-4309-b63f-5d321670891e:] ~ graph rollback due to exception AtlasBaseException:Instance hive_db with unique attribute {qualifiedName=aem_develop@C03} does not exist (GraphTransactionInterceptor:73) could you please let me know what i have to do next to solve this issue. Thanks&Regards, Owez ... View more

@Sarath Subramanian Below are the jars present in /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrth total 123M -rw-r--r-- 1 atlas hadoop 4.4M May 11 2018 scala-reflect-2.11.8.jar -rw-r--r-- 1 atlas hadoop 785K May 11 2018 scalap-2.11.8.jar -rw-r--r-- 1 atlas hadoop 5.5M May 11 2018 scala-library-2.11.8.jar -rw-r--r-- 1 atlas hadoop 15M May 11 2018 scala-compiler-2.11.8.jar -rw-r--r-- 1 atlas hadoop 1.6M May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 7.2M May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 67K May 11 2018 json4s-native_2.11-3.2.11.jar -rw-r--r-- 1 atlas hadoop 576K May 11 2018 json4s-core_2.11-3.2.11.jar -rw-r--r-- 1 atlas hadoop 81K May 11 2018 json4s-ast_2.11-3.2.11.jar -rw-r--r-- 1 atlas hadoop 53K May 11 2018 jersey-multipart-1.19.jar -rw-r--r-- 1 atlas hadoop 69K May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 12K May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 564K May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 228K May 11 2018 gson-2.5.jar -rw-r--r-- 1 atlas hadoop 400K May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 96K May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 217K May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 64K May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 18K May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 43K May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 atlas hadoop 35K May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar -rw-r--r-- 1 root root 141K Nov 29 11:19 commons-digester.jar -rw-r--r-- 1 root root 141K Nov 29 11:19 commons-digester-1.8.jar -rw-r--r-- 1 atlas hadoop 292K Nov 29 11:19 commons-configuration.jar -rw-r--r-- 1 atlas hadoop 292K Nov 29 11:19 commons-configuration-1.6.jar -rw-r--r-- 1 root root 236K Nov 29 11:19 commons-compress.jar -rw-r--r-- 1 root root 236K Nov 29 11:19 commons-compress-1.4.1.jar -rw-r--r-- 1 root root 575K Nov 29 11:19 commons-collections.jar -rw-r--r-- 1 root root 575K Nov 29 11:19 commons-collections-3.2.2.jar -rw-r--r-- 1 root root 57K Nov 29 11:19 commons-codec.jar -rw-r--r-- 1 root root 57K Nov 29 11:19 commons-codec-1.4.jar -rw-r--r-- 1 root root 41K Nov 29 11:19 commons-cli.jar -rw-r--r-- 1 root root 41K Nov 29 11:19 commons-cli-1.2.jar -rw-r--r-- 1 root root 185K Nov 29 11:19 commons-beanutils.jar -rw-r--r-- 1 root root 202K Nov 29 11:19 commons-beanutils-core.jar -rw-r--r-- 1 root root 202K Nov 29 11:19 commons-beanutils-core-1.8.0.jar -rw-r--r-- 1 root root 185K Nov 29 11:19 commons-beanutils-1.7.0.jar -rw-r--r-- 1 root root 768K Nov 29 11:19 azure-storage.jar -rw-r--r-- 1 root root 768K Nov 29 11:19 azure-storage-5.4.0.jar -rw-r--r-- 1 root root 9.9K Nov 29 11:19 azure-keyvault-core.jar -rw-r--r-- 1 root root 9.9K Nov 29 11:19 azure-keyvault-core-0.8.0.jar -rw-r--r-- 1 root root 297K Nov 29 11:19 avro.jar -rw-r--r-- 1 root root 297K Nov 29 11:19 avro-1.7.4.jar -rw-r--r-- 1 root root 79K Nov 29 11:19 api-util.jar -rw-r--r-- 1 root root 79K Nov 29 11:19 api-util-1.0.0-M20.jar -rw-r--r-- 1 root root 17K Nov 29 11:19 api-asn1-api.jar -rw-r--r-- 1 root root 17K Nov 29 11:19 api-asn1-api-1.0.0-M20.jar -rw-r--r-- 1 root root 676K Nov 29 11:19 apacheds-kerberos-codec.jar -rw-r--r-- 1 root root 676K Nov 29 11:19 apacheds-kerberos-codec-2.0.0-M15.jar -rw-r--r-- 1 root root 44K Nov 29 11:19 apacheds-i18n.jar -rw-r--r-- 1 root root 44K Nov 29 11:19 apacheds-i18n-2.0.0-M15.jar -rw-r--r-- 1 root root 62K Nov 29 11:19 activation.jar -rw-r--r-- 1 root root 62K Nov 29 11:19 activation-1.1.jar -rw-r--r-- 1 root root 40K Nov 29 11:19 hadoop-mapreduce-client-jobclient.jar -rw-r--r-- 1 root root 40K Nov 29 11:19 hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 1.5M Nov 29 11:19 hadoop-mapreduce-client-core.jar -rw-r--r-- 1 root root 1.5M Nov 29 11:19 hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 737K Nov 29 11:19 hadoop-mapreduce-client-common.jar -rw-r--r-- 1 root root 737K Nov 29 11:19 hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 509K Nov 29 11:19 hadoop-mapreduce-client-app.jar -rw-r--r-- 1 root root 509K Nov 29 11:19 hadoop-mapreduce-client-app-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 8.4M Nov 29 11:19 hadoop-hdfs.jar -rw-r--r-- 1 root root 8.4M Nov 29 11:19 hadoop-hdfs-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 3.6M Nov 29 11:19 hadoop-common.jar -rw-r--r-- 1 root root 3.6M Nov 29 11:19 hadoop-common-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 78K Nov 29 11:19 hadoop-auth.jar -rw-r--r-- 1 root root 78K Nov 29 11:19 hadoop-auth-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 17K Nov 29 11:19 hadoop-annotations.jar -rw-r--r-- 1 root root 17K Nov 29 11:19 hadoop-annotations-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 1.6M Nov 29 11:19 guava.jar -rw-r--r-- 1 root root 1.6M Nov 29 11:19 guava-11.0.2.jar -rw-r--r-- 1 root root 186K Nov 29 11:19 gson.jar -rw-r--r-- 1 root root 186K Nov 29 11:19 gson-2.2.4.jar -rw-r--r-- 1 root root 265K Nov 29 11:19 curator-recipes.jar -rw-r--r-- 1 root root 265K Nov 29 11:19 curator-recipes-2.7.1.jar -rw-r--r-- 1 root root 182K Nov 29 11:19 curator-framework.jar -rw-r--r-- 1 root root 182K Nov 29 11:19 curator-framework-2.7.1.jar -rw-r--r-- 1 root root 68K Nov 29 11:19 curator-client.jar -rw-r--r-- 1 root root 68K Nov 29 11:19 curator-client-2.7.1.jar -rw-r--r-- 1 root root 267K Nov 29 11:19 commons-net.jar -rw-r--r-- 1 root root 267K Nov 29 11:19 commons-net-3.1.jar -rw-r--r-- 1 root root 1.6M Nov 29 11:19 commons-math3.jar -rw-r--r-- 1 root root 1.6M Nov 29 11:19 commons-math3-3.1.1.jar -rw-r--r-- 1 root root 61K Nov 29 11:19 commons-logging.jar -rw-r--r-- 1 root root 61K Nov 29 11:19 commons-logging-1.1.3.jar -rw-r--r-- 1 root root 278K Nov 29 11:19 commons-lang.jar -rw-r--r-- 1 root root 425K Nov 29 11:19 commons-lang3.jar -rw-r--r-- 1 root root 425K Nov 29 11:19 commons-lang3-3.4.jar -rw-r--r-- 1 root root 278K Nov 29 11:19 commons-lang-2.6.jar -rw-r--r-- 1 root root 181K Nov 29 11:19 commons-io.jar -rw-r--r-- 1 root root 181K Nov 29 11:19 commons-io-2.4.jar -rw-r--r-- 1 root root 27K Nov 29 11:19 jackson-xc.jar -rw-r--r-- 1 root root 27K Nov 29 11:19 jackson-xc-1.9.13.jar -rw-r--r-- 1 root root 763K Nov 29 11:19 jackson-mapper-asl.jar -rw-r--r-- 1 root root 763K Nov 29 11:19 jackson-mapper-asl-1.9.13.jar -rw-r--r-- 1 root root 18K Nov 29 11:19 jackson-jaxrs.jar -rw-r--r-- 1 root root 18K Nov 29 11:19 jackson-jaxrs-1.9.13.jar -rw-r--r-- 1 root root 846K Nov 29 11:19 jackson-databind.jar -rw-r--r-- 1 root root 846K Nov 29 11:19 jackson-databind-2.2.3.jar -rw-r--r-- 1 root root 189K Nov 29 11:19 jackson-core.jar -rw-r--r-- 1 root root 227K Nov 29 11:19 jackson-core-asl.jar -rw-r--r-- 1 root root 227K Nov 29 11:19 jackson-core-asl-1.9.13.jar -rw-r--r-- 1 root root 189K Nov 29 11:19 jackson-core-2.2.3.jar -rw-r--r-- 1 root root 33K Nov 29 11:19 jackson-annotations.jar -rw-r--r-- 1 root root 33K Nov 29 11:19 jackson-annotations-2.2.3.jar -rw-r--r-- 1 root root 320K Nov 29 11:19 httpcore.jar -rw-r--r-- 1 root root 320K Nov 29 11:19 httpcore-4.4.4.jar -rw-r--r-- 1 root root 720K Nov 29 11:19 httpclient.jar -rw-r--r-- 1 root root 720K Nov 29 11:19 httpclient-4.5.2.jar -rw-r--r-- 1 root root 1.5M Nov 29 11:19 htrace-core.jar -rw-r--r-- 1 root root 1.5M Nov 29 11:19 htrace-core-3.1.0-incubating.jar -rw-r--r-- 1 root root 414K Nov 29 11:19 hadoop-yarn-server-common.jar -rw-r--r-- 1 root root 414K Nov 29 11:19 hadoop-yarn-server-common-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 100K Nov 29 11:19 hadoop-yarn-registry.jar -rw-r--r-- 1 root root 100K Nov 29 11:19 hadoop-yarn-registry-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 1.8M Nov 29 11:19 hadoop-yarn-common.jar -rw-r--r-- 1 root root 1.8M Nov 29 11:19 hadoop-yarn-common-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 198K Nov 29 11:19 hadoop-yarn-client.jar -rw-r--r-- 1 root root 198K Nov 29 11:19 hadoop-yarn-client-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 2.1M Nov 29 11:19 hadoop-yarn-api.jar -rw-r--r-- 1 root root 2.1M Nov 29 11:19 hadoop-yarn-api-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 54K Nov 29 11:19 hadoop-mapreduce-client-shuffle.jar -rw-r--r-- 1 root root 54K Nov 29 11:19 hadoop-mapreduce-client-shuffle-2.7.3.2.6.5.0-292.jar -rw-r--r-- 1 root root 1.2M Nov 29 11:19 netty.jar -rw-r--r-- 1 root root 2.2M Nov 29 11:19 netty-all.jar -rw-r--r-- 1 root root 2.2M Nov 29 11:19 netty-all-4.0.52.Final.jar -rw-r--r-- 1 root root 1.2M Nov 29 11:19 netty-3.6.2.Final.jar -rw-r--r-- 1 root root 479K Nov 29 11:19 log4j.jar -rw-r--r-- 1 root root 479K Nov 29 11:19 log4j-1.2.17.jar -rw-r--r-- 1 root root 1022K Nov 29 11:19 leveldbjni-all.jar -rw-r--r-- 1 root root 1022K Nov 29 11:19 leveldbjni-all-1.8.jar -rw-r--r-- 1 root root 33K Nov 29 11:19 jsr305.jar -rw-r--r-- 1 root root 33K Nov 29 11:19 jsr305-3.0.0.jar -rw-r--r-- 1 root root 99K Nov 29 11:19 jsp-api.jar -rw-r--r-- 1 root root 99K Nov 29 11:19 jsp-api-2.1.jar -rw-r--r-- 1 root root 77K Nov 29 11:19 json-smart.jar -rw-r--r-- 1 root root 77K Nov 29 11:19 json-smart-1.3.1.jar -rw-r--r-- 1 root root 174K Nov 29 11:19 jetty-util.jar -rw-r--r-- 1 root root 174K Nov 29 11:19 jetty-util-6.1.26.hwx.jar -rw-r--r-- 1 root root 19K Nov 29 11:19 jetty-sslengine.jar -rw-r--r-- 1 root root 19K Nov 29 11:19 jetty-sslengine-6.1.26.hwx.jar -rw-r--r-- 1 root root 448K Nov 29 11:19 jersey-core.jar -rw-r--r-- 1 root root 448K Nov 29 11:19 jersey-core-1.9.jar -rw-r--r-- 1 root root 128K Nov 29 11:19 jersey-client.jar -rw-r--r-- 1 root root 128K Nov 29 11:19 jersey-client-1.9.jar -rw-r--r-- 1 root root 4.7K Nov 29 11:19 jcip-annotations.jar -rw-r--r-- 1 root root 4.7K Nov 29 11:19 jcip-annotations-1.0-1.jar -rw-r--r-- 1 root root 103K Nov 29 11:19 jaxb-api.jar -rw-r--r-- 1 root root 103K Nov 29 11:19 jaxb-api-2.2.2.jar -rw-r--r-- 1 root root 15K Nov 29 11:20 xmlenc.jar -rw-r--r-- 1 root root 15K Nov 29 11:20 xmlenc-0.52.jar -rw-r--r-- 1 root root 190K Nov 29 11:20 xml-apis.jar -rw-r--r-- 1 root root 190K Nov 29 11:20 xml-apis-1.3.04.jar -rw-r--r-- 1 root root 1.2M Nov 29 11:20 xercesImpl.jar -rw-r--r-- 1 root root 1.2M Nov 29 11:20 xercesImpl-2.9.1.jar -rw-r--r-- 1 root root 23K Nov 29 11:20 stax-api.jar -rw-r--r-- 1 root root 23K Nov 29 11:20 stax-api-1.0-2.jar -rw-r--r-- 1 root root 973K Nov 29 11:20 snappy-java.jar -rw-r--r-- 1 root root 973K Nov 29 11:20 snappy-java-1.0.4.1.jar -rw-r--r-- 1 root root 8.7K Nov 29 11:20 slf4j-log4j12.jar -rw-r--r-- 1 root root 8.7K Nov 29 11:20 slf4j-log4j12-1.7.10.jar -rw-r--r-- 1 root root 32K Nov 29 11:20 slf4j-api.jar -rw-r--r-- 1 root root 32K Nov 29 11:20 slf4j-api-1.7.10.jar -rw-r--r-- 1 root root 103K Nov 29 11:20 servlet-api.jar -rw-r--r-- 1 root root 103K Nov 29 11:20 servlet-api-2.5.jar -rw-r--r-- 1 root root 521K Nov 29 11:20 protobuf-java.jar -rw-r--r-- 1 root root 521K Nov 29 11:20 protobuf-java-2.5.0.jar -rw-r--r-- 1 root root 29K Nov 29 11:20 paranamer.jar -rw-r--r-- 1 root root 29K Nov 29 11:20 paranamer-2.3.jar -rw-r--r-- 1 root root 65K Nov 29 11:20 okio.jar -rw-r--r-- 1 root root 65K Nov 29 11:20 okio-1.6.0.jar -rw-r--r-- 1 root root 324K Nov 29 11:20 okhttp.jar -rw-r--r-- 1 root root 324K Nov 29 11:20 okhttp-2.7.5.jar -rw-r--r-- 1 root root 293K Nov 29 11:20 nimbus-jose-jwt.jar -rw-r--r-- 1 root root 293K Nov 29 11:20 nimbus-jose-jwt-4.41.1.jar -rw-r--r-- 1 root root 785K Nov 29 11:20 zookeeper.jar -rw-r--r-- 1 root root 785K Nov 29 11:20 zookeeper-3.4.6.2.6.5.0-292.jar -rw-r--r-- 1 root root 93K Nov 29 11:20 xz.jar -rw-r--r-- 1 root root 93K Nov 29 11:20 xz-1.0.jar root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# pwd /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt |wc -l 180 Could you please let me know which jars are missing or any sym links are missing. Below is the current issue while running import-hive.sh script. -rwxr-xr-x 1 atlas hadoop 3652 May 11 2018 import-hive.sh* root@hadwkrndcc03-3:/usr/hdp/2.6.5.0-292/atlas/hook-bin# ./import-hive.sh Using Hive configuration directory [/etc/hive/conf] Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/atlas/hook/hive/atlas-hive-plugin-impl/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/atlas/hook/hive/atlas-hive-plugin-impl/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hive/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout. log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout. Enter username for atlas :- admin Enter password for atlas :- Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jettison/json/JSONException at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:124) Caused by: java.lang.ClassNotFoundException: org.codehaus.jettison.json.JSONException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more Failed to import Hive Meta Data!!! root@hadwkrndcc03-3:/usr/hdp/2.6.5.0-292/atlas/hook-bin# Thanks&Regards, Owez Mujawar ... View more

@Sarath Subramanian Please find below issue while publishing message to ATLAS_ENTITIES. atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-producer.sh --broker-list hadwkrndcc03-3.lifeway.org:6667 --topic ATLAS_ENTITIES >olla >[2018-11-29 05:50:15,694] ERROR Error when sending message to topic ATLAS_ENTITIES with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 1526 ms has passed since batch creation plus linger time Could you please help in this issue and provide a solution for it. Thanks in advance. Regards, Owez Mujawar ... View more

@Sarath Subramanian Hi, As we are trying to execute the query but we are failing. below error for the same. could you please help me to resolved this issue. atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/current/atlas-server/bin/quick_start.py Enter username for atlas :- admin Enter password for atlas :- Creating sample types: Created type [DB] Created type [Table] Created type [StorageDesc] Created type [Column] Created type [LoadProcess] Created type [View] Created type [JdbcAccess] Created type [ETL] Created type [Metric] Created type [PII] Created type [Fact] Created type [Dimension] Created type [Log Data] Creating sample entities: Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClientV2$API_V2@514646ef failed with status 500 (Internal Server Error) Response Body ({"errorCode":"ATLAS-500-00-007","errorMessage":"Failed to notify NotificationEntityChangeListener for change CREATE"}) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:387) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:320) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:208) at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:281) at org.apache.atlas.examples.QuickStartV2.createInstance(QuickStartV2.java:339) at org.apache.atlas.examples.QuickStartV2.createDatabase(QuickStartV2.java:362) at org.apache.atlas.examples.QuickStartV2.createEntities(QuickStartV2.java:268) at org.apache.atlas.examples.QuickStartV2.runQuickstart(QuickStartV2.java:150) at org.apache.atlas.examples.QuickStartV2.main(QuickStartV2.java:132) No sample data added to Apache Atlas Server. Thanks in advance. Regards, Owez Mujawar ... View more

@Sarath Subramanian As we have copied commons-configuration.jar is given path. /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl then Again we have tried to run the script but still issue persist. /usr/hdp/2.6.5.0-292/atlas/hook-bin# ./import-hive.sh Using Hive configuration directory [/etc/hive/conf] Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout. log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout. Enter username for atlas :- admin Enter password for atlas :- Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/ClientHandlerException at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:124) Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.ClientHandlerException at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more Failed to import Hive Meta Data!!! Could you please help me with this. Thanks in Advance. Regards, Owez Mujawar ... View more