Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hive insert query taking so much of time

avatar

I have created a simple table and trying to insert data but it is taking too much of time , even more than 5 min.

create table command :

hive> create table poc(id int);
OK
Time taken: 1.578 seconds

but when try to insert data it is taking so much of time :

hive> create table poc(id int);
OK
Time taken: 1.578 seconds
hive> insert into  poc values(1);
Query ID = hive_20180915064819_83183eef-8dcc-463a-872e-fd8c58453af5
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1536988895253_0006)

--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1                 INITED      1          0        0        1       0       0
--------------------------------------------------------------------------------
VERTICES: 00/01  [>>--------------------------] 0%    ELAPSED TIME: 159.96 s   
  • execution engine : Tez
  • Tez container size : 5120 MB
  • Memory allocated for all YARN containers on a node : 15GB
  • Minimum Container Size (Memory) : 1024 MB
  • Maximum Container Size (Memory) : 15GB
  • Single node cluster

snap of RM UI:

Apps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory UsedMemory TotalMemory ReservedVCores UsedVCores TotalVCores ReservedActive NodesDecommissioned NodesLost NodesUnhealthy NodesRebooted Nodes
712426 GB15 GB0 B26010000
6 REPLIES 6

avatar
Master Guru

@Anurag Mishra

Probably the session is initializing when you are inserting the data for the first time.

After first insert(session is already initialized), did u tried to insert more values into the same table and check does those inserts also taking 5 mins or not.

avatar

but why does it take too much of time , even this is the first insert , is there something wrong with the memory tuning ?

avatar
Expert Contributor

Is it MR job ? what is the value for hive.execution.engine? Are you using specific queue to launch this job ? It seems to be resource unavailable issue. If AppMaster is launched, can you collect application_1536988895253_0006 logs & check why task container's are not getting launched yet.

avatar

thanks for your reply ,

I have already mentioned in the post value of hive.execution.engine is tez , also as it is taking more time definitely it seems to be resource issue . I am more curious about how to best tune hive with given configurations.

avatar
Contributor

hi @Anurag Mishra,

for memeory : https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

did you tried in uber mode ?

<code>mapreduce.job.ubertask.enable = true

avatar
Super Collaborator

Hi @Anurag Mishra,

It seems Tez unable to launch the session. first kill the all running applications and retry to lauch the job. if it doesn't work tune the tez configuration seetting by using below url :

https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html