Member since
08-28-2017
26
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2691 | 03-06-2019 03:20 PM | |
5812 | 12-03-2018 05:52 AM |
03-06-2019
03:20 PM
Hi Team, We have found the solution. it was not related to Atlas. Issue was user limit. Recently increase in HDFS data causes more processing at jobs level. Current Hiveserver2 thread count was not sufficient to handle these increased amount of processing. This causes thread creation error for Hive service as shown below: Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717) After stopping Atlas service also we were getting same error. To resolve this error, we have increased thread creation capacity of Hive service by increasing value of 'hive_user_nproc_limit' parameter for hive user. Increased in HDFS data might be an coincidence with addition of Atlas service. Regards, Owez Mujawar
... View more
02-11-2019
05:57 PM
Hi Team, We have enabled Atlas in our environment. After that we started getting connection hanging to Hive/HS2/Knox Gateway. Hiveserver doesn't create new thread connections and stops responding. when we do restart of HS2 then the new jobs start but after sometime again connection goes in hung state.This issue was a result of recently configured atlas service on the cluster. After that followed below link and changed the configuration. https://community.hortonworks.com/content/supportkb/148579/failed-hive-internal-error-javautilconcurrentrejec.html atlas.hook.hive.maxThreads=50 from 5. atlas.hook.hive.minThreads=5. Even i checked my user limit for hive it has been set: root@hostname:~# su - hive
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 64
process 16000
nofiles 32000
vmemory(kbytes) unlimited
locks unlimited Logs for the same: 2019-02-07 06:00:02,911 INFO [HiveServer2-HttpHandler-Pool: Thread-346402]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=PostHook.org.apache.atlas.hive.hook.HiveHook from=org.apache.hadoop.hive.ql.Driver>2019-02-07 06:00:02,911 ERROR [HiveServer2-HttpHandler-Pool: Thread-346402]: hook.HiveHook (HiveHook.java:run(213)) - Submitting to thread pool failed due to errorjava.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64ee3a83 rejected from java.util.concurrent.ThreadPoolExecutor@3f70557d[Running, pool size = 5, active threads = 5, queued tasks = 1000, completed tasks = 1704] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:193) at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:52) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1599) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289) Could you please help me with this. Thanks in Advance. Regards, Owez Mujawar
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hadoop
-
Apache Hive
12-20-2018
09:59 AM
@Vinay Could you please check your ambari-agent and ambari-server version. check in database as well. This issue occurs when there are incorrect entries present in the ambari database. The following is the table in the database.
From the host_version
ambari=>select*from host_version; Thanks, Owez
... View more
12-17-2018
03:16 PM
@Sharmadha Sainath Please find attachment for your reference. We have offsets.topic.replication.factor=1 and we have 1 Kafka broker. We haven't found any error in Kafka logs. KAFKA Logs: [2018-12-13 04:29:43,286] INFO Updated PartitionLeaderEpoch. New: {epoch:16, offset:1063289}, Current: {epoch:14, offset916036} for Partition: ATLAS_ENTITIES-0. Cache now contains 8 entries. (kafka.server.epoch.LeaderEpochFileCache)
[2018-12-13 04:34:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-12-13 04:44:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2018-12-13 04:54:11,764] INFO [GroupMetadataManager brokerId=1004] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager) Still we haven't fount any solution. Could you please help me to resolved this issue on priority. Thanks, Owez
... View more
12-13-2018
02:09 PM
@Sharmadha Sainath Hive Hook is enabled Please find attachment for your reference. Checked hive permission and Atlas that is perfect no issues in Permission. Still when we are creating new tables it is not reflecting in Atlas ui. we need to run import-hive.sh. We are getting below error when we run import-hive.sh script. it is coming in application.log in Atlas rest services has no errors in their logs. This errors are for newly created table. but still due to this error also we are able to see external table and its lineage. 2018-12-13 04:32:58,136 ERROR - [pool-2-thread-9 - 04e4ba41-c09d-4257-9391-1fde800fa5d2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep13@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:32:59,132 ERROR - [pool-2-thread-10 - fd8d6366-e465-450a-a42a-8d98022787f2:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.dep2@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:01,145 ERROR - [pool-2-thread-9 - bceb4f09-1cce-461d-8b1a-2e4a90e39717:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.dep2@C03:1544606069000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:04,374 ERROR - [pool-2-thread-7 - e62c07bc-4fe4-480a-bf80-6ffff789c698:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee19@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:04,975 ERROR - [pool-2-thread-8 - d2f31e6c-fe99-40d8-a153-087b9c10eb8b:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee20@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:05,347 ERROR - [pool-2-thread-9 - 4f478d9e-4c32-456e-8669-0d4bcef9bcf8:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data0@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:06,139 ERROR - [pool-2-thread-9 - e6da522f-ff85-436b-b66b-2844ca2fdf2c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data04@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:06,388 ERROR - [pool-2-thread-8 - a385f421-d818-4fc6-a613-880064babcf9:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data04@C03:1544688356000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:07,468 ERROR - [pool-2-thread-10 - e5c8525c-7877-4e1a-b2e6-b23d9f054e94:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data18@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:07,959 ERROR - [pool-2-thread-9 - 72143f28-2630-4306-8639-7c513dffc51c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_process with unique attribute {qualifiedName=default.employee_data2@C03:1544605910000} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:08,180 ERROR - [pool-2-thread-10 - 516cb986-69ae-48f9-813f-3e7b23a1b967:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data20@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-13 04:33:08,439 ERROR - [pool-2-thread-9 - 5706eb89-2951-4c33-b914-9ace101cb011:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=default.employee_data3@C03} does not exist (GraphTransactionInterceptor:73) After running import script we are able to see only External tables in Atlas. no lineage for Managed and View tables. Could you please help me on this. Thanks, Owez
... View more
12-12-2018
01:15 PM
@Sharmadha Sainath Thanks for the confirmation. The scenario you explain that's perfect suits in our environment, but 3rd Scenario is not working in our environment. for example we are using below query to create Managed,View and External table: CREATE TABLE IF NOT EXISTS employee13 ( eid int, name int) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/hive/data/employee13';
CREATE VIEW employee_data AS SELECT * FROM employee WHERE eid >0;
CREATE EXTERNAL TABLE employee_data2 ( eid int, name int ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n' LOCATION '/hive/data/employee'; When we execute above query its gets successfully executed. we don't see any errors in logs. When we login to Atlas UI we are not able to see that newly created tables in 1st Attempt. After running import-hive.sh script we are able to see the lineage of External tables only. We are not able see lineage of newly created managed and view tables.it comes as no linage found. Could you please guide me on that.
... View more
12-11-2018
01:24 PM
@Sharmadha Sainath Hive Hook is enabled and we can see all tables and linage of External table only. We want to see lineage of Managed and Virtual view tables which were created before atlas. Could you please help me to see that lineage of Managed and Virtual view tables. i have not seen any error in ranger-log , kafka-log , and hive-log. only in application-log i can see below error. 2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73)
2018-12-11 00:11:19,247 INFO - [main-SendThread(hadmgrndcc03-1.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-1.lifeway.org/172.17.20.29:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019)
2018-12-11 00:11:30,951 INFO - [main-SendThread(hadmgrndcc03-3.lifeway.org:2181):] ~ Opening socket connection to server hadmgrndcc03-3.lifeway.org/172.17.20.33:2181. Will not attempt to authenticate using SASL (unknown error) (ClientCnxn:1019)
2018-12-11 00:12:48,690 ERROR - [pool-2-thread-7 - 3b35712a-6c84-4102-a19c-cedf26ea5e1c:] ~ graph rollback due to exception AtlasBaseException:Instance __AtlasUserProfile with unique attribute {name=admin} does not exist (GraphTransactionInterceptor:73) Let me know how we can see lineage of Managed and virtual view tables.
... View more
12-10-2018
07:02 PM
@Ronak bansal We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 We are getting below error while running import-hive.sh script. root@hn:/var/log/atlas# tail -f application.log
2018-12-10 04:50:06,783 ERROR - [pool-2-thread-7 - d35c5859-a93e-40ce-94ff-5c12e45f2880:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_awood@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:07,100 ERROR - [pool-2-thread-7 - 0b130511-ed2a-4434-95ad-692b0babf348:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:07,431 ERROR - [pool-2-thread-7 - fbf49dd4-12fd-4ff1-bdf7-d69187ead26c:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.dim_type2_new@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:07,826 ERROR - [pool-2-thread-7 - 1041077f-a6cf-4b24-8669-d84d88155d82:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:08,056 ERROR - [pool-2-thread-9 - ce6d8709-db1a-458e-8cfb-4d1ccbb28ca6:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_cdc@C03} does not exist (GraphTransactionInterceptor:73)
2018-12-10 04:50:08,332 ERROR - [pool-2-thread-7 - 1757d3a0-4562-4c9b-bf1b-526cd34b2c05:] ~ graph rollback due to exception AtlasBaseException:Instance hive_table with unique attribute {qualifiedName=work.type2_current@C03} does not exist (GraphTransactionInterceptor:73)
In Atlas we are able to see External tables linage but not for Managed and Virtual Views tables. Could you please let me why we are getting this error. Can we see linage for Managed and Virtual view tables? Thanks, Owez
... View more
12-10-2018
06:57 PM
@Sharmadha Sainath @Ana Gillan Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Questions: 1) We need to know can we see linage of Managed tables and virtual views linage in Atlas? 2) When we create a new tables and try to see linage of that tables in first attempt we are not able to see linage of that tables after running import-hive.sh script we are able to see the linage of that newly created tables, why this is happening? it should show automatically the linage of newly created tables. 3) Could you please let me know is there any property need to set or any configuration needs to change. Two snap shot has been attached 1) atlas_external table as we can see linage of that table after running import-hive.sh script. 2) atlas_managed table we cannot see linage of that table even after running script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables and virtual views as well? There is no error in application.log, hbase-log, ranger-audit, kafka-log. Permission has been set in ranger. Please suggest. We are stuck now. Thanks, Owez
... View more
12-07-2018
01:14 PM
Hi Team, We are using HDP-2.6.5.0 and Ambari-2.6.2.0, Atlas-0.8.0 Now we are able to see linage of older tables as well, but only for External tables. As when we are trying to create a new managed tables and when we create a joins/intersecting two tables, we are not able to see their linage. if we create a new tables as external tables whereas we can see the linage of that tables after running import-hive.sh script. Request you to let me know that can we see linage for external tables only? or we can see linage for managed tables as well? @team Please let me know what we can do in Atlas and what we cannot do in Atlas.
... View more