- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
hive insert query taking so much of time
- Labels:
-
Apache Hive
-
Apache Tez
-
Apache YARN
Created ‎09-15-2018 06:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have created a simple table and trying to insert data but it is taking too much of time , even more than 5 min.
create table command :
hive> create table poc(id int); OK Time taken: 1.578 seconds
but when try to insert data it is taking so much of time :
hive> create table poc(id int); OK Time taken: 1.578 seconds hive> insert into poc values(1); Query ID = hive_20180915064819_83183eef-8dcc-463a-872e-fd8c58453af5 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1536988895253_0006) -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 INITED 1 0 0 1 0 0 -------------------------------------------------------------------------------- VERTICES: 00/01 [>>--------------------------] 0% ELAPSED TIME: 159.96 s
- execution engine : Tez
- Tez container size : 5120 MB
- Memory allocated for all YARN containers on a node : 15GB
- Minimum Container Size (Memory) : 1024 MB
- Maximum Container Size (Memory) : 15GB
- Single node cluster
snap of RM UI:
Apps Submitted | Apps Pending | Apps Running | Apps Completed | Containers Running | Memory Used | Memory Total | Memory Reserved | VCores Used | VCores Total | VCores Reserved | Active Nodes | Decommissioned Nodes | Lost Nodes | Unhealthy Nodes | Rebooted Nodes |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
7 | 1 | 2 | 4 | 2 | 6 GB | 15 GB | 0 B | 2 | 6 | 0 | 1 | 0 | 0 | 0 | 0 |
Created ‎09-15-2018 11:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Probably the session is initializing when you are inserting the data for the first time.
After first insert(session is already initialized), did u tried to insert more values into the same table and check does those inserts also taking 5 mins or not.
Created ‎09-16-2018 08:59 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
but why does it take too much of time , even this is the first insert , is there something wrong with the memory tuning ?
Created ‎09-16-2018 02:36 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is it MR job ? what is the value for hive.execution.engine? Are you using specific queue to launch this job ? It seems to be resource unavailable issue. If AppMaster is launched, can you collect application_1536988895253_0006 logs & check why task container's are not getting launched yet.
Created ‎09-16-2018 03:41 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thanks for your reply ,
I have already mentioned in the post value of hive.execution.engine is tez , also as it is taking more time definitely it seems to be resource issue . I am more curious about how to best tune hive with given configurations.
Created ‎09-17-2018 06:16 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hi @Anurag Mishra,
for memeory : https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
did you tried in uber mode ?
<code>mapreduce.job.ubertask.enable = true
Created ‎09-17-2018 06:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Anurag Mishra,
It seems Tez unable to launch the session. first kill the all running applications and retry to lauch the job. if it doesn't work tune the tez configuration seetting by using below url :
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html
