Member since
08-28-2017
26
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1946 | 03-06-2019 03:20 PM | |
3885 | 12-03-2018 05:52 AM |
03-29-2019
05:19 AM
Hi @Anpan It is better to have one AD for each environment. You can control each user with setting up a different principal's like hdfs principal for DEV users and separate for PROD users. You can easy maintained different OU and Groups for each Environment as per requirement in your organisation. It is also recommended to have only one AD server for each environment. This will be cost effective and easy to manage OU , Groups and Princiapls etc.. Thanks, Owez
... View more
03-29-2019
05:04 AM
Hi Balaji, Could you please switch your user to hdfs with "hdpuser" then try to put the files on hdfs. And also could you please check your hdfs policy in Ranger. Thanks, Owez
... View more
03-06-2019
03:20 PM
Hi Team, We have found the solution. it was not related to Atlas. Issue was user limit. Recently increase in HDFS data causes more processing at jobs level. Current Hiveserver2 thread count was not sufficient to handle these increased amount of processing. This causes thread creation error for Hive service as shown below: Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717) After stopping Atlas service also we were getting same error. To resolve this error, we have increased thread creation capacity of Hive service by increasing value of 'hive_user_nproc_limit' parameter for hive user. Increased in HDFS data might be an coincidence with addition of Atlas service. Regards, Owez Mujawar
... View more
02-19-2019
02:54 PM
@Ahmad Hajitorab , Try to run sample metadata in Atlas. su atlas -c '/usr/hdp/current/atlas-server/bin/quick_start.py' after then check application.log. Questions:- Have you enabled Plugin's and Have you set Audit to Solr? You can also run import-hive.sh. Please share your logs details.
... View more
02-11-2019
05:57 PM
Hi Team, We have enabled Atlas in our environment. After that we started getting connection hanging to Hive/HS2/Knox Gateway. Hiveserver doesn't create new thread connections and stops responding. when we do restart of HS2 then the new jobs start but after sometime again connection goes in hung state.This issue was a result of recently configured atlas service on the cluster. After that followed below link and changed the configuration. https://community.hortonworks.com/content/supportkb/148579/failed-hive-internal-error-javautilconcurrentrejec.html atlas.hook.hive.maxThreads=50 from 5. atlas.hook.hive.minThreads=5. Even i checked my user limit for hive it has been set: root@hostname:~# su - hive
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 64
process 16000
nofiles 32000
vmemory(kbytes) unlimited
locks unlimited Logs for the same: 2019-02-07 06:00:02,911 INFO [HiveServer2-HttpHandler-Pool: Thread-346402]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook from=org.apache.hadoop.hive.ql.Driver>2019-02-07 06:00:02,911 ERROR [HiveServer2-HttpHandler-Pool: Thread-346402]: hook.HiveHook (HiveHook.java:run(213)) - Submitting to thread pool failed due to errorjava.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64ee3a83 rejected from java.util.concurrent.ThreadPoolExecutor@3f70557d[Running, pool size = 5, active threads = 5, queued tasks = 1000, completed tasks = 1704] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:193) at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1599) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) Could you please help me with this. Thanks in Advance. Regards, Owez Mujawar
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hadoop
-
Apache Hive
12-20-2018
09:59 AM
@Vinay Could you please check your ambari-agent and ambari-server version. check in database as well. This issue occurs when there are incorrect entries present in the ambari database. The following is the table in the database.
From the host_version
ambari=>select*from host_version; Thanks, Owez
... View more
12-19-2018
06:57 AM
@Kishore Jannu Could you try below solution. it has worked for me. mv /var/lib/ambari-agent/data/structured-out-status.json /var/lib/ambari-agent/data/structured-out-status.json-old Try to restart ambari-server restart ambari-agent restart Or Could you please check your value for dfs.permissions.superusergroup. This might be causing your issue. Try to modify the dfs.permissions.superusergroup variable from hdfs. Could you please let me know if issue is still persist. Thanks, Owez
... View more
12-17-2018
03:16 PM
@Sharmadha Sainath Please find attachment for your reference. We have offsets.topic.replication.factor=1 and we have 1 Kafka broker. We haven't found any error in Kafka logs. KAFKA Logs: [2018-12-13 04:29:43,286] INFO Updated PartitionLeaderEpoch. New: {epoch:16, offset:1063289}, Current: {epoch:14, offset916036} for Partition: ATLAS_ENTITIES-0. Cache now contains 8 entries. (kafka.server.epoch.LeaderEpochFileCache)
[2018-12-13 04:34:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-12-13 04:44:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-12-13 04:54:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) Still we haven't fount any solution. Could you please help me to resolved this issue on priority. Thanks, Owez
... View more
12-13-2018
02:09 PM
@Sharmadha Sainath Hive Hook is enabled Please find attachment for your reference. Checked hive permission and Atlas that is perfect no issues in Permission. Still when we are creating new tables it is not reflecting in Atlas ui. we need to run import-hive.sh. We are getting below error when we run import-hive.sh script. it is coming in application.log in Atlas rest services has no errors in their logs. This errors are for newly created table. but still due to this error also we are able to see external table and its lineage. 2018-12-13 04:32:58,136 ERROR - [pool-2-thread-9 - 04e4ba41-c09d-4257-9391-1fde800fa5d2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep13@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:32:59,132 ERROR - [pool-2-thread-10 - fd8d6366-e465-450a-a42a-8d98022787f2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep2@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:01,145 ERROR - [pool-2-thread-9 - bceb4f09-1cce-461d-8b1a-2e4a90e39717:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.dep2@C03:1544606069000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:04,374 ERROR - [pool-2-thread-7 - e62c07bc-4fe4-480a-bf80-6ffff789c698:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee19@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:04,975 ERROR - [pool-2-thread-8 - d2f31e6c-fe99-40d8-a153-087b9c10eb8b:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee20@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:05,347 ERROR - [pool-2-thread-9 - 4f478d9e-4c32-456e-8669-0d4bcef9bcf8:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data0@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:06,139 ERROR - [pool-2-thread-9 - e6da522f-ff85-436b-b66b-2844ca2fdf2c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data04@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:06,388 ERROR - [pool-2-thread-8 - a385f421-d818-4fc6-a613-880064babcf9:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data04@C03:1544688356000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:07,468 ERROR - [pool-2-thread-10 - e5c8525c-7877-4e1a-b2e6-b23d9f054e94:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data18@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:07,959 ERROR - [pool-2-thread-9 - 72143f28-2630-4306-8639-7c513dffc51c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data2@C03:1544605910000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:08,180 ERROR - [pool-2-thread-10 - 516cb986-69ae-48f9-813f-3e7b23a1b967:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data20@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:08,439 ERROR - [pool-2-thread-9 - 5706eb89-2951-4c33-b914-9ace101cb011:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data3@C03} does not exist (GraphTransactionInterceptor:73) After running import script we are able to see only External tables in Atlas. no lineage for Managed and View tables. Could you please help me on this. Thanks, Owez
... View more
12-12-2018
01:15 PM
@Sharmadha Sainath Thanks for the confirmation. The scenario you explain that's perfect suits in our environment, but 3rd Scenario is not working in our environment. for example we are using below query to create Managed,View and External table: CREATE TABLE IF NOT EXISTS employee13 ( eid int, name int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/hive/data/employee13';
CREATE VIEW employee_data AS SELECT * FROM employee WHERE eid >0;
CREATE EXTERNAL TABLE employee_data2 ( eid int, name int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOCATION '/hive/data/employee'; When we execute above query its gets successfully executed. we don't see any errors in logs. When we login to Atlas UI we are not able to see that newly created tables in 1st Attempt. After running import-hive.sh script we are able to see the lineage of External tables only. We are not able see lineage of newly created managed and view tables.it comes as no linage found. Could you please guide me on that.
... View more
12-11-2018
01:24 PM
@Sharmadha Sainath Hive Hook is enabled and we can see all tables and linage of External table only. We want to see lineage of Managed and Virtual view tables which were created before atlas. Could you please help me to see that lineage of Managed and Virtual view tables. i have not seen any error in ranger-log , kafka-log , and hive-log. only in application-log i can see below error. 2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73)
2018-12-11 00:11:19,247 INFO - [main-SendThread(hadmgrndcc03-1.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-1.lifeway.org/172.17.20.29:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019)
2018-12-11 00:11:30,951 INFO - [main-SendThread(hadmgrndcc03-3.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-3.lifeway.org/172.17.20.33:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019)
2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73) Let me know how we can see lineage of Managed and virtual view tables.
... View more
12-10-2018
07:02 PM
@Ronak bansal We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 We are getting below error while running import-hive.sh script. root@hn:/var/log/atlas# tail -f application.log
2018-12-10 04:50:06,783 ERROR - [pool-2-thread-7 - d35c5859-a93e-40ce-94ff-5c12e45f2880:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_awood@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:07,100 ERROR - [pool-2-thread-7 - 0b130511-ed2a-4434-95ad-692b0babf348:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:07,431 ERROR - [pool-2-thread-7 - fbf49dd4-12fd-4ff1-bdf7-d69187ead26c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2_new@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:07,826 ERROR - [pool-2-thread-7 - 1041077f-a6cf-4b24-8669-d84d88155d82:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:08,056 ERROR - [pool-2-thread-9 - ce6d8709-db1a-458e-8cfb-4d1ccbb28ca6:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_cdc@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:08,332 ERROR - [pool-2-thread-7 - 1757d3a0-4562-4c9b-bf1b-526cd34b2c05:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_current@C03} does not exist (GraphTransactionInterceptor:73)
In Atlas we are able to see External tables linage but not for Managed and Virtual Views tables. Could you please let me why we are getting this error. Can we see linage for Managed and Virtual view tables? Thanks, Owez
... View more
12-10-2018
06:57 PM
@Sharmadha Sainath @Ana Gillan Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Questions: 1) We need to know can we see linage of Managed tables and virtual views linage in Atlas? 2) When we create a new tables and try to see linage of that tables in first attempt we are not able to see linage of that tables after running import-hive.sh script we are able to see the linage of that newly created tables, why this is happening? it should show automatically the linage of newly created tables. 3) Could you please let me know is there any property need to set or any configuration needs to change. Two snap shot has been attached 1) atlas_external table as we can see linage of that table after running import-hive.sh script. 2) atlas_managed table we cannot see linage of that table even after running script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables and virtual views as well? There is no error in application.log, hbase-log, ranger-audit, kafka-log. Permission has been set in ranger. Please suggest. We are stuck now. Thanks, Owez
... View more
12-10-2018
06:06 AM
@artmro Could you please check the logs for Atlas, hbase and ranger-audit. You will find the error. please post that error. i have resolved that issue. Actually my issue was for jersey-jar issue. Let us know the error logs so that we can assist you on the same. Thanks, Owez
... View more
12-07-2018
01:14 PM
Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Now we are able to see linage of older tables as well, but only for External tables. As when we are trying to create a new managed tables and when we create a joins/intersecting two tables, we are not able to see their linage. if we create a new tables as external tables whereas we can see the linage of that tables after running import-hive.sh script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables as well? @team Please let me know what we can do in Atlas and what we cannot do in Atlas.
... View more
12-06-2018
03:00 PM
Hello Team , We are using HDP-2.6.5.0 and Ambari-2.6.2.0. We have executed import-hive.sh script to import already existing hive tables in Atlas. It got successfully executed. Now we can see all hive databases and tables in Atlas, but we are not able to see data linage of that imported tables. If we create external table on any hdfs path then we can see lineage in Atlas. Also, If we create any managed tables in Atlas, we are not able to see lineage of that tables. Why we are not getting lineage of older tables and new managed tables? Please suggest. We are stuck now. Thanks, Owez
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hive
12-05-2018
06:54 AM
@Arshadullah Khan Even i have faced same issue Actually it doesn't work. i think this is a bug. As we should connect to at least one hive server when HA is configured. Bug is in Knox Gateway.
... View more
12-05-2018
06:32 AM
@Arshadullah Khan Thanks for the suggestion issue got resolved. Downloaded and Copied jar to /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl and issue got resolved after running import-hive.sh script. Now we are able to see Hive_db and Hive_tables in Atlas Web UI. Thanks & regards, Owez
... View more
12-04-2018
02:03 PM
@Sarath Subramanian yes i have copied some of the jars which were not there in below path, but the list you share that jars are already present in path. /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl -rw-r--r-- 1 atlas hadoop 4573750 May 11 2018 scala-reflect-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scalap-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 802818 May 11 2018 scalap-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scala-library-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 5744974 May 11 2018 scala-library-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt scala-compiler-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 15487351 May 11 2018 scala-compiler-2.11.8.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt kafka-clients-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 1590487 May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt kafka_2.11-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 7532355 May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-native_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 67829 May 11 2018 json4s-native_2.11-3.2.11.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-core_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 589462 May 11 2018 json4s-core_2.11-3.2.11.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt json4s-ast_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 82421 May 11 2018 json4s-ast_2.11-3.2.11.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt jersey-multipart-1.19.jar
-rw-r--r-- 1 atlas hadoop 53275 May 11 2018 jersey-multipart-1.19.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hive-bridge-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 70540 May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hdfs-model-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 11639 May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt hbase-common-1.1.2.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 577458 May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt gson-2.5.jar
-rw-r--r-- 1 atlas hadoop 232482 May 11 2018 gson-2.5.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-typesystem-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 408745 May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-notification-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 98220 May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-intg-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 222091 May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 64777 May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-v2-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 18039 May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-v1-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 43413 May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar
root@hostname:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt atlas-client-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 35271 May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar
in application.log below is the error 2018-12-04 03:54:47,186 ERROR - [pool-2-thread-9 - 138daf6f-4389-4309-b63f-5d321670891e:] ~ graph rollback due to exception AtlasBaseException:Instance hive_db with unique attribute {qualifiedName=aem_develop@C03} does not exist (GraphTransactionInterceptor:73) could you please let me know what i have to do next to solve this issue. Thanks&Regards, Owez
... View more
12-03-2018
01:51 PM
@Sarath Subramanian Below are the jars present in /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrth
total 123M
-rw-r--r-- 1 atlas hadoop 4.4M May 11 2018 scala-reflect-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 785K May 11 2018 scalap-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 5.5M May 11 2018 scala-library-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 15M May 11 2018 scala-compiler-2.11.8.jar
-rw-r--r-- 1 atlas hadoop 1.6M May 11 2018 kafka-clients-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 7.2M May 11 2018 kafka_2.11-1.0.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 67K May 11 2018 json4s-native_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 576K May 11 2018 json4s-core_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 81K May 11 2018 json4s-ast_2.11-3.2.11.jar
-rw-r--r-- 1 atlas hadoop 53K May 11 2018 jersey-multipart-1.19.jar
-rw-r--r-- 1 atlas hadoop 69K May 11 2018 hive-bridge-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 12K May 11 2018 hdfs-model-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 564K May 11 2018 hbase-common-1.1.2.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 228K May 11 2018 gson-2.5.jar
-rw-r--r-- 1 atlas hadoop 400K May 11 2018 atlas-typesystem-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 96K May 11 2018 atlas-notification-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 217K May 11 2018 atlas-intg-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 64K May 11 2018 atlas-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 18K May 11 2018 atlas-client-v2-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 43K May 11 2018 atlas-client-v1-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 atlas hadoop 35K May 11 2018 atlas-client-common-0.8.0.2.6.5.0-292.jar
-rw-r--r-- 1 root root 141K Nov 29 11:19 commons-digester.jar
-rw-r--r-- 1 root root 141K Nov 29 11:19 commons-digester-1.8.jar
-rw-r--r-- 1 atlas hadoop 292K Nov 29 11:19 commons-configuration.jar
-rw-r--r-- 1 atlas hadoop 292K Nov 29 11:19 commons-configuration-1.6.jar
-rw-r--r-- 1 root root 236K Nov 29 11:19 commons-compress.jar
-rw-r--r-- 1 root root 236K Nov 29 11:19 commons-compress-1.4.1.jar
-rw-r--r-- 1 root root 575K Nov 29 11:19 commons-collections.jar
-rw-r--r-- 1 root root 575K Nov 29 11:19 commons-collections-3.2.2.jar
-rw-r--r-- 1 root root 57K Nov 29 11:19 commons-codec.jar
-rw-r--r-- 1 root root 57K Nov 29 11:19 commons-codec-1.4.jar
-rw-r--r-- 1 root root 41K Nov 29 11:19 commons-cli.jar
-rw-r--r-- 1 root root 41K Nov 29 11:19 commons-cli-1.2.jar
-rw-r--r-- 1 root root 185K Nov 29 11:19 commons-beanutils.jar
-rw-r--r-- 1 root root 202K Nov 29 11:19 commons-beanutils-core.jar
-rw-r--r-- 1 root root 202K Nov 29 11:19 commons-beanutils-core-1.8.0.jar
-rw-r--r-- 1 root root 185K Nov 29 11:19 commons-beanutils-1.7.0.jar
-rw-r--r-- 1 root root 768K Nov 29 11:19 azure-storage.jar
-rw-r--r-- 1 root root 768K Nov 29 11:19 azure-storage-5.4.0.jar
-rw-r--r-- 1 root root 9.9K Nov 29 11:19 azure-keyvault-core.jar
-rw-r--r-- 1 root root 9.9K Nov 29 11:19 azure-keyvault-core-0.8.0.jar
-rw-r--r-- 1 root root 297K Nov 29 11:19 avro.jar
-rw-r--r-- 1 root root 297K Nov 29 11:19 avro-1.7.4.jar
-rw-r--r-- 1 root root 79K Nov 29 11:19 api-util.jar
-rw-r--r-- 1 root root 79K Nov 29 11:19 api-util-1.0.0-M20.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 api-asn1-api.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 api-asn1-api-1.0.0-M20.jar
-rw-r--r-- 1 root root 676K Nov 29 11:19 apacheds-kerberos-codec.jar
-rw-r--r-- 1 root root 676K Nov 29 11:19 apacheds-kerberos-codec-2.0.0-M15.jar
-rw-r--r-- 1 root root 44K Nov 29 11:19 apacheds-i18n.jar
-rw-r--r-- 1 root root 44K Nov 29 11:19 apacheds-i18n-2.0.0-M15.jar
-rw-r--r-- 1 root root 62K Nov 29 11:19 activation.jar
-rw-r--r-- 1 root root 62K Nov 29 11:19 activation-1.1.jar
-rw-r--r-- 1 root root 40K Nov 29 11:19 hadoop-mapreduce-client-jobclient.jar
-rw-r--r-- 1 root root 40K Nov 29 11:19 hadoop-mapreduce-client-jobclient-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 hadoop-mapreduce-client-core.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 hadoop-mapreduce-client-core-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 737K Nov 29 11:19 hadoop-mapreduce-client-common.jar
-rw-r--r-- 1 root root 737K Nov 29 11:19 hadoop-mapreduce-client-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 509K Nov 29 11:19 hadoop-mapreduce-client-app.jar
-rw-r--r-- 1 root root 509K Nov 29 11:19 hadoop-mapreduce-client-app-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 8.4M Nov 29 11:19 hadoop-hdfs.jar
-rw-r--r-- 1 root root 8.4M Nov 29 11:19 hadoop-hdfs-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 3.6M Nov 29 11:19 hadoop-common.jar
-rw-r--r-- 1 root root 3.6M Nov 29 11:19 hadoop-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 78K Nov 29 11:19 hadoop-auth.jar
-rw-r--r-- 1 root root 78K Nov 29 11:19 hadoop-auth-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 hadoop-annotations.jar
-rw-r--r-- 1 root root 17K Nov 29 11:19 hadoop-annotations-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 guava.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 guava-11.0.2.jar
-rw-r--r-- 1 root root 186K Nov 29 11:19 gson.jar
-rw-r--r-- 1 root root 186K Nov 29 11:19 gson-2.2.4.jar
-rw-r--r-- 1 root root 265K Nov 29 11:19 curator-recipes.jar
-rw-r--r-- 1 root root 265K Nov 29 11:19 curator-recipes-2.7.1.jar
-rw-r--r-- 1 root root 182K Nov 29 11:19 curator-framework.jar
-rw-r--r-- 1 root root 182K Nov 29 11:19 curator-framework-2.7.1.jar
-rw-r--r-- 1 root root 68K Nov 29 11:19 curator-client.jar
-rw-r--r-- 1 root root 68K Nov 29 11:19 curator-client-2.7.1.jar
-rw-r--r-- 1 root root 267K Nov 29 11:19 commons-net.jar
-rw-r--r-- 1 root root 267K Nov 29 11:19 commons-net-3.1.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 commons-math3.jar
-rw-r--r-- 1 root root 1.6M Nov 29 11:19 commons-math3-3.1.1.jar
-rw-r--r-- 1 root root 61K Nov 29 11:19 commons-logging.jar
-rw-r--r-- 1 root root 61K Nov 29 11:19 commons-logging-1.1.3.jar
-rw-r--r-- 1 root root 278K Nov 29 11:19 commons-lang.jar
-rw-r--r-- 1 root root 425K Nov 29 11:19 commons-lang3.jar
-rw-r--r-- 1 root root 425K Nov 29 11:19 commons-lang3-3.4.jar
-rw-r--r-- 1 root root 278K Nov 29 11:19 commons-lang-2.6.jar
-rw-r--r-- 1 root root 181K Nov 29 11:19 commons-io.jar
-rw-r--r-- 1 root root 181K Nov 29 11:19 commons-io-2.4.jar
-rw-r--r-- 1 root root 27K Nov 29 11:19 jackson-xc.jar
-rw-r--r-- 1 root root 27K Nov 29 11:19 jackson-xc-1.9.13.jar
-rw-r--r-- 1 root root 763K Nov 29 11:19 jackson-mapper-asl.jar
-rw-r--r-- 1 root root 763K Nov 29 11:19 jackson-mapper-asl-1.9.13.jar
-rw-r--r-- 1 root root 18K Nov 29 11:19 jackson-jaxrs.jar
-rw-r--r-- 1 root root 18K Nov 29 11:19 jackson-jaxrs-1.9.13.jar
-rw-r--r-- 1 root root 846K Nov 29 11:19 jackson-databind.jar
-rw-r--r-- 1 root root 846K Nov 29 11:19 jackson-databind-2.2.3.jar
-rw-r--r-- 1 root root 189K Nov 29 11:19 jackson-core.jar
-rw-r--r-- 1 root root 227K Nov 29 11:19 jackson-core-asl.jar
-rw-r--r-- 1 root root 227K Nov 29 11:19 jackson-core-asl-1.9.13.jar
-rw-r--r-- 1 root root 189K Nov 29 11:19 jackson-core-2.2.3.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jackson-annotations.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jackson-annotations-2.2.3.jar
-rw-r--r-- 1 root root 320K Nov 29 11:19 httpcore.jar
-rw-r--r-- 1 root root 320K Nov 29 11:19 httpcore-4.4.4.jar
-rw-r--r-- 1 root root 720K Nov 29 11:19 httpclient.jar
-rw-r--r-- 1 root root 720K Nov 29 11:19 httpclient-4.5.2.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 htrace-core.jar
-rw-r--r-- 1 root root 1.5M Nov 29 11:19 htrace-core-3.1.0-incubating.jar
-rw-r--r-- 1 root root 414K Nov 29 11:19 hadoop-yarn-server-common.jar
-rw-r--r-- 1 root root 414K Nov 29 11:19 hadoop-yarn-server-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 100K Nov 29 11:19 hadoop-yarn-registry.jar
-rw-r--r-- 1 root root 100K Nov 29 11:19 hadoop-yarn-registry-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.8M Nov 29 11:19 hadoop-yarn-common.jar
-rw-r--r-- 1 root root 1.8M Nov 29 11:19 hadoop-yarn-common-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 198K Nov 29 11:19 hadoop-yarn-client.jar
-rw-r--r-- 1 root root 198K Nov 29 11:19 hadoop-yarn-client-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 2.1M Nov 29 11:19 hadoop-yarn-api.jar
-rw-r--r-- 1 root root 2.1M Nov 29 11:19 hadoop-yarn-api-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 54K Nov 29 11:19 hadoop-mapreduce-client-shuffle.jar
-rw-r--r-- 1 root root 54K Nov 29 11:19 hadoop-mapreduce-client-shuffle-2.7.3.2.6.5.0-292.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:19 netty.jar
-rw-r--r-- 1 root root 2.2M Nov 29 11:19 netty-all.jar
-rw-r--r-- 1 root root 2.2M Nov 29 11:19 netty-all-4.0.52.Final.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:19 netty-3.6.2.Final.jar
-rw-r--r-- 1 root root 479K Nov 29 11:19 log4j.jar
-rw-r--r-- 1 root root 479K Nov 29 11:19 log4j-1.2.17.jar
-rw-r--r-- 1 root root 1022K Nov 29 11:19 leveldbjni-all.jar
-rw-r--r-- 1 root root 1022K Nov 29 11:19 leveldbjni-all-1.8.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jsr305.jar
-rw-r--r-- 1 root root 33K Nov 29 11:19 jsr305-3.0.0.jar
-rw-r--r-- 1 root root 99K Nov 29 11:19 jsp-api.jar
-rw-r--r-- 1 root root 99K Nov 29 11:19 jsp-api-2.1.jar
-rw-r--r-- 1 root root 77K Nov 29 11:19 json-smart.jar
-rw-r--r-- 1 root root 77K Nov 29 11:19 json-smart-1.3.1.jar
-rw-r--r-- 1 root root 174K Nov 29 11:19 jetty-util.jar
-rw-r--r-- 1 root root 174K Nov 29 11:19 jetty-util-6.1.26.hwx.jar
-rw-r--r-- 1 root root 19K Nov 29 11:19 jetty-sslengine.jar
-rw-r--r-- 1 root root 19K Nov 29 11:19 jetty-sslengine-6.1.26.hwx.jar
-rw-r--r-- 1 root root 448K Nov 29 11:19 jersey-core.jar
-rw-r--r-- 1 root root 448K Nov 29 11:19 jersey-core-1.9.jar
-rw-r--r-- 1 root root 128K Nov 29 11:19 jersey-client.jar
-rw-r--r-- 1 root root 128K Nov 29 11:19 jersey-client-1.9.jar
-rw-r--r-- 1 root root 4.7K Nov 29 11:19 jcip-annotations.jar
-rw-r--r-- 1 root root 4.7K Nov 29 11:19 jcip-annotations-1.0-1.jar
-rw-r--r-- 1 root root 103K Nov 29 11:19 jaxb-api.jar
-rw-r--r-- 1 root root 103K Nov 29 11:19 jaxb-api-2.2.2.jar
-rw-r--r-- 1 root root 15K Nov 29 11:20 xmlenc.jar
-rw-r--r-- 1 root root 15K Nov 29 11:20 xmlenc-0.52.jar
-rw-r--r-- 1 root root 190K Nov 29 11:20 xml-apis.jar
-rw-r--r-- 1 root root 190K Nov 29 11:20 xml-apis-1.3.04.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:20 xercesImpl.jar
-rw-r--r-- 1 root root 1.2M Nov 29 11:20 xercesImpl-2.9.1.jar
-rw-r--r-- 1 root root 23K Nov 29 11:20 stax-api.jar
-rw-r--r-- 1 root root 23K Nov 29 11:20 stax-api-1.0-2.jar
-rw-r--r-- 1 root root 973K Nov 29 11:20 snappy-java.jar
-rw-r--r-- 1 root root 973K Nov 29 11:20 snappy-java-1.0.4.1.jar
-rw-r--r-- 1 root root 8.7K Nov 29 11:20 slf4j-log4j12.jar
-rw-r--r-- 1 root root 8.7K Nov 29 11:20 slf4j-log4j12-1.7.10.jar
-rw-r--r-- 1 root root 32K Nov 29 11:20 slf4j-api.jar
-rw-r--r-- 1 root root 32K Nov 29 11:20 slf4j-api-1.7.10.jar
-rw-r--r-- 1 root root 103K Nov 29 11:20 servlet-api.jar
-rw-r--r-- 1 root root 103K Nov 29 11:20 servlet-api-2.5.jar
-rw-r--r-- 1 root root 521K Nov 29 11:20 protobuf-java.jar
-rw-r--r-- 1 root root 521K Nov 29 11:20 protobuf-java-2.5.0.jar
-rw-r--r-- 1 root root 29K Nov 29 11:20 paranamer.jar
-rw-r--r-- 1 root root 29K Nov 29 11:20 paranamer-2.3.jar
-rw-r--r-- 1 root root 65K Nov 29 11:20 okio.jar
-rw-r--r-- 1 root root 65K Nov 29 11:20 okio-1.6.0.jar
-rw-r--r-- 1 root root 324K Nov 29 11:20 okhttp.jar
-rw-r--r-- 1 root root 324K Nov 29 11:20 okhttp-2.7.5.jar
-rw-r--r-- 1 root root 293K Nov 29 11:20 nimbus-jose-jwt.jar
-rw-r--r-- 1 root root 293K Nov 29 11:20 nimbus-jose-jwt-4.41.1.jar
-rw-r--r-- 1 root root 785K Nov 29 11:20 zookeeper.jar
-rw-r--r-- 1 root root 785K Nov 29 11:20 zookeeper-3.4.6.2.6.5.0-292.jar
-rw-r--r-- 1 root root 93K Nov 29 11:20 xz.jar
-rw-r--r-- 1 root root 93K Nov 29 11:20 xz-1.0.jar
root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl#
root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# pwd
/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl
root@hadwkrndcc03-3:/usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl# ls -lrt |wc -l
180 Could you please let me know which jars are missing or any sym links are missing. Below is the current issue while running import-hive.sh script. -rwxr-xr-x 1 atlas hadoop 3652 May 11 2018 import-hive.sh*
root@hadwkrndcc03-3:/usr/hdp/2.6.5.0-292/atlas/hook-bin# ./import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/atlas/hook/hive/atlas-hive-plugin-impl/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/atlas/hook/hive/atlas-hive-plugin-impl/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hive/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout.
Enter username for atlas :- admin
Enter password for atlas :-
Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/jettison/json/JSONException
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:124)
Caused by: java.lang.ClassNotFoundException: org.codehaus.jettison.json.JSONException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
Failed to import Hive Meta Data!!!
root@hadwkrndcc03-3:/usr/hdp/2.6.5.0-292/atlas/hook-bin# Thanks&Regards, Owez Mujawar
... View more
12-03-2018
05:52 AM
@Sarath Subramanian now i am able publish a sample message to ATLAS_ENTITIES. Resolution: Deleted the older Atlas_entities and created a new one. Thanks for the help. Regards, Owez Mujawar
... View more
11-29-2018
03:06 PM
@Sarath Subramanian Please find below issue while publishing message to ATLAS_ENTITIES. atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-producer.sh --broker-list hadwkrndcc03-3.lifeway.org:6667 --topic ATLAS_ENTITIES
>olla
>[2018-11-29 05:50:15,694] ERROR Error when sending message to topic ATLAS_ENTITIES with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 1526 ms has passed since batch creation plus linger time Could you please help in this issue and provide a solution for it. Thanks in advance. Regards, Owez Mujawar
... View more
11-29-2018
02:20 PM
@Sarath Subramanian Hi, As we are trying to execute the query but we are failing. below error for the same. could you please help me to resolved this issue. atlas@hadwkrndcc03-3:/var/log/atlas$ /usr/hdp/current/atlas-server/bin/quick_start.py
Enter username for atlas :- admin
Enter password for atlas :-
Creating sample types:
Created type [DB]
Created type [Table]
Created type [StorageDesc]
Created type [Column]
Created type [LoadProcess]
Created type [View]
Created type [JdbcAccess]
Created type [ETL]
Created type [Metric]
Created type [PII]
Created type [Fact]
Created type [Dimension]
Created type [Log Data]
Creating sample entities:
Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClientV2$API_V2@514646ef failed with status 500 (Internal Server Error) Response Body ({"errorCode":"ATLAS-500-00-007","errorMessage":"Failed to notify NotificationEntityChangeListener for change CREATE"})
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:387)
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:320)
at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:208)
at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:281)
at org.apache.atlas.examples.QuickStartV2.createInstance(QuickStartV2.java:339)
at org.apache.atlas.examples.QuickStartV2.createDatabase(QuickStartV2.java:362)
at org.apache.atlas.examples.QuickStartV2.createEntities(QuickStartV2.java:268)
at org.apache.atlas.examples.QuickStartV2.runQuickstart(QuickStartV2.java:150)
at org.apache.atlas.examples.QuickStartV2.main(QuickStartV2.java:132)
No sample data added to Apache Atlas Server. Thanks in advance. Regards, Owez Mujawar
... View more
11-29-2018
02:19 PM
@Sarath Subramanian As we have copied commons-configuration.jar is given path. /usr/hdp/current/atlas-server/hook/hive/atlas-hive-plugin-impl then Again we have tried to run the script but still issue persist. /usr/hdp/2.6.5.0-292/atlas/hook-bin# ./import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout.
Enter username for atlas :- admin
Enter password for atlas :-
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/ClientHandlerException
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:124)
Caused by: java.lang.ClassNotFoundException: com.sun.jersey.api.client.ClientHandlerException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 1 more
Failed to import Hive Meta Data!!! Could you please help me with this. Thanks in Advance. Regards, Owez Mujawar
... View more
11-28-2018
03:06 PM
Hi team, As i am trying to run import-hive.sh script but it is not able to excute it. below are the logs attached. /usr/hdp/2.6.5.0-292/atlas/hook-bin/import-hive.sh
Using Hive configuration directory [/etc/hive/conf]
Log file for import is /usr/hdp/2.6.5.0-292/atlas/logs/import-hive.log
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/PropertiesConfiguration
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:112)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.PropertiesConfiguration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
Failed to import Hive Meta Data!!! Could you please help me with this. Thanks in Advance. Regards, Owez Mujawar
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hadoop
-
Apache Hive
11-28-2018
03:02 PM
Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0.
While creating new entity in Atlas we are getting following error in application.log. Currently we are not using Ranger. we trying to create new entity without ranger. Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:220)
at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:182)
at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:117)
at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:208)
at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:102)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityC
hangeNotifier.java:176)
... 100 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94)
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:64)
at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29)
at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:207)
... 105 more
Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ATLAS_ENTITIES-0: 30009 ms has passed since batch creation plus linger time
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ ==> releaseLockedObjects(): lockedGuids.size: 0 (GraphTransactionInterceptor:187)
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ <== releaseLockedObjects(): lockedGuids.size: 0 (GraphTransactionInterceptor:197)
2018-11-28 01:35:02,638 DEBUG - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ PERF|EntityREST.createOrUpdate()|30228 (EntityREST:77)
2018-11-28 01:35:02,638 ERROR - [pool-2-thread-10 - 2b585636-30dd-4484-a6a6-e4c05ae50167:] ~ Error handling a request: 168f59a001543090 (ExceptionMapperUtil:32)
org.apache.atlas.exception.AtlasBaseException: Failed to notify NotificationEntityChangeListener for change UPDATE
at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:183)
I can describe topic for ATLAS_ENTITIES /usr/hdp/2.6.5.0-292/kafka/bin/kafka-topics.sh --describe --topic ATLAS_ENTITIES --zookeeper hadmgrndcc03-1.lifeway.org:2181,hadmgrndcc03-2.lifeway.org:2181,hadmgrndcc03-3.lifeway.org:2181
Topic:ATLAS_ENTITIES PartitionCount:1 ReplicationFactor:1 Configs:
Topic: ATLAS_ENTITIES Partition: 0 Leader: 1003 Replicas: 1003 Isr: 1003
ACL Permission:-
/usr/hdp/2.6.5.0-292/kafka/bin/kafka-acls.sh --add --allow-principal user:* --producer --topic ATLAS_ENTITIES --authorizer-properties zookeeper.connect=hadmgrndcc03-1.lifeway.org:2181,hadmgrndcc03-2.lifeway.org:2181,hadmgrndcc03-3.lifeway.org:2181
Adding ACLs for resource `Topic:ATLAS_ENTITIES`:
user:* has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Describe from hosts: *
Adding ACLs for resource `Cluster:kafka-cluster`:
user:* has Allow permission for operations: Create from hosts: *
Current ACLs for resource `Topic:ATLAS_ENTITIES`:
User: has Allow permission for operations: Read from hosts: *
User: has Allow permission for operations: Describe from hosts: *
user:* has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Describe from hosts: *
User:* has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Write from hosts: application.log
User:* has Allow permission for operations: Describe from hosts: *
User:atlas has Allow permission for operations: Write from hosts: *
user:* has Allow permission for operations: Read from hosts: application.log
User:* has Allow permission for operations: Read from hosts: *
User:atlas has Allow permission for operations: Describe from hosts: * Could you please help in this issue and provide a solution for it. Thanks in advance. Regards,
Owez Mujawar
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Kafka