Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Please see the Cloudera blog for information on the Cloudera Response to CVE-2021-4428

Restrict memory for multiple hive query

Explorer

Hi All,

We have total of 3.5 TB of RAM and We are facing a problem of memory utilization .

Problem description :

We have 5 HQL (Each HQL has multiple sub query ) running in parallel .

One of the query is occupying 3.3 TB of memory and rest are sitting idle in the queue and taking lot of time to complete .

We need some recommendation or tuning so that all queries gets equal chunk of memory .

Parameters set in our query

SET hive.exec.compress.output=true ;SET hive.exec.compress.intermediate=true ;
SET mapred.output.compress=true ;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec ;
SET mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec ;
SET io.seqfile.compression.type=BLOCK;
SET io.sort.mb=500 ;
SET dfs.block.size=536870912;
SET io.file.buffer.size=131072;
SET mapred.compress.map.output=true;
SET mapred.output.compression.type=BLOCK;
SET hive.auto.convert.join=true;
SET mapreduce.map.memory.mb=12288;
SET mapreduce.map.java.opts =-Xmx9831m;
SET mapreduce.reduce.java.opts=-Xmx8192m;
SET mapreduce.reduce.memory.mb=10240;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.allow-drop-table=true;

Attached screen shot as well for reference .

memory.jpg

Quick Help is appreciated !!

1 REPLY 1

Hi ,

 

Where you able to get this issue sorted out.

 

Regards

Sam