Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hive insert query taking so much of time

Highlighted

hive insert query taking so much of time

I have created a simple table and trying to insert data but it is taking too much of time , even more than 5 min.

create table command :

hive> create table poc(id int);
OK
Time taken: 1.578 seconds

but when try to insert data it is taking so much of time :

hive> create table poc(id int);
OK
Time taken: 1.578 seconds
hive> insert into  poc values(1);
Query ID = hive_20180915064819_83183eef-8dcc-463a-872e-fd8c58453af5
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id application_1536988895253_0006)

--------------------------------------------------------------------------------
        VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED
--------------------------------------------------------------------------------
Map 1                 INITED      1          0        0        1       0       0
--------------------------------------------------------------------------------
VERTICES: 00/01  [>>--------------------------] 0%    ELAPSED TIME: 159.96 s   
  • execution engine : Tez
  • Tez container size : 5120 MB
  • Memory allocated for all YARN containers on a node : 15GB
  • Minimum Container Size (Memory) : 1024 MB
  • Maximum Container Size (Memory) : 15GB
  • Single node cluster

snap of RM UI:

Apps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory UsedMemory TotalMemory ReservedVCores UsedVCores TotalVCores ReservedActive NodesDecommissioned NodesLost NodesUnhealthy NodesRebooted Nodes
712426 GB15 GB0 B26010000
6 REPLIES 6
Highlighted

Re: hive insert query taking so much of time

Super Guru

@Anurag Mishra

Probably the session is initializing when you are inserting the data for the first time.

After first insert(session is already initialized), did u tried to insert more values into the same table and check does those inserts also taking 5 mins or not.

Highlighted

Re: hive insert query taking so much of time

but why does it take too much of time , even this is the first insert , is there something wrong with the memory tuning ?

Re: hive insert query taking so much of time

Expert Contributor

Is it MR job ? what is the value for hive.execution.engine? Are you using specific queue to launch this job ? It seems to be resource unavailable issue. If AppMaster is launched, can you collect application_1536988895253_0006 logs & check why task container's are not getting launched yet.

Highlighted

Re: hive insert query taking so much of time

thanks for your reply ,

I have already mentioned in the post value of hive.execution.engine is tez , also as it is taking more time definitely it seems to be resource issue . I am more curious about how to best tune hive with given configurations.

Highlighted

Re: hive insert query taking so much of time

Contributor

hi @Anurag Mishra,

for memeory : https://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/

did you tried in uber mode ?

<code>mapreduce.job.ubertask.enable = true
Highlighted

Re: hive insert query taking so much of time

Super Collaborator

Hi @Anurag Mishra,

It seems Tez unable to launch the session. first kill the all running applications and retry to lauch the job. if it doesn't work tune the tez configuration seetting by using below url :

https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html

Don't have an account?
Coming from Hortonworks? Activate your account here