Created 09-15-2018 06:57 AM
I have created a simple table and trying to insert data but it is taking too much of time , even more than 5 min.
create table command :
hive> create table poc(id int); OK Time taken: 1.578 seconds
but when try to insert data it is taking so much of time :
hive> create table poc(id int); OK Time taken: 1.578 seconds hive> insert into poc values(1); Query ID = hive_20180915064819_83183eef-8dcc-463a-872e-fd8c58453af5 Total jobs = 1 Launching Job 1 out of 1 Status: Running (Executing on YARN cluster with App id application_1536988895253_0006) -------------------------------------------------------------------------------- VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED -------------------------------------------------------------------------------- Map 1 INITED 1 0 0 1 0 0 -------------------------------------------------------------------------------- VERTICES: 00/01 [>>--------------------------] 0% ELAPSED TIME: 159.96 s
snap of RM UI:
Apps Submitted | Apps Pending | Apps Running | Apps Completed | Containers Running | Memory Used | Memory Total | Memory Reserved | VCores Used | VCores Total | VCores Reserved | Active Nodes | Decommissioned Nodes | Lost Nodes | Unhealthy Nodes | Rebooted Nodes |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
7 | 1 | 2 | 4 | 2 | 6 GB | 15 GB | 0 B | 2 | 6 | 0 | 1 | 0 | 0 | 0 | 0 |
Created 09-15-2018 11:54 AM
Probably the session is initializing when you are inserting the data for the first time.
After first insert(session is already initialized), did u tried to insert more values into the same table and check does those inserts also taking 5 mins or not.
Created 09-16-2018 08:59 AM
but why does it take too much of time , even this is the first insert , is there something wrong with the memory tuning ?
Created 09-16-2018 02:36 PM
Is it MR job ? what is the value for hive.execution.engine? Are you using specific queue to launch this job ? It seems to be resource unavailable issue. If AppMaster is launched, can you collect application_1536988895253_0006 logs & check why task container's are not getting launched yet.
Created 09-16-2018 03:41 PM
thanks for your reply ,
I have already mentioned in the post value of hive.execution.engine is tez , also as it is taking more time definitely it seems to be resource issue . I am more curious about how to best tune hive with given configurations.
Created 09-17-2018 06:16 AM
hi @Anurag Mishra,
for memeory : https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/
did you tried in uber mode ?
<code>mapreduce.job.ubertask.enable = true
Created 09-17-2018 06:26 AM
Hi @Anurag Mishra,
It seems Tez unable to launch the session. first kill the all running applications and retry to lauch the job. if it doesn't work tune the tez configuration seetting by using below url :
https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html