Hi All, We have total of 3.5 TB of RAM and We are facing a problem of memory utilization . Problem description : We have 5 HQL (Each HQL has multiple sub query ) running in parallel . One of the query is occupying 3.3 TB of memory and rest are sitting idle in the queue and taking lot of time to complete . We need some recommendation or tuning so that all queries gets equal chunk of memory . Parameters set in our query SET hive.exec.compress.output=true ;SET hive.exec.compress.intermediate=true ; SET mapred.output.compress=true ; SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec ; SET mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec ; SET io.seqfile.compression.type=BLOCK; SET io.sort.mb=500 ; SET dfs.block.size=536870912; SET io.file.buffer.size=131072; SET mapred.compress.map.output=true; SET mapred.output.compression.type=BLOCK; SET hive.auto.convert.join=true; SET mapreduce.map.memory.mb=12288; SET mapreduce.map.java.opts =-Xmx9831m; SET mapreduce.reduce.java.opts=-Xmx8192m; SET mapreduce.reduce.memory.mb=10240; SET hive.exec.dynamic.partition.mode=nonstrict; SET hive.allow-drop-table=true; Attached screen shot as well for reference . memory.jpg Quick Help is appreciated !! ... View more

Hi All, When i am ruining hive quries some times i am facing error: Insufficient space for shared memory file . After few tries it throws Exception in thread "main" java.io.IOException: No space left on device Help Appreciated . ... View more

Hi All, Is there any way we can skip to the next row based on some filter conditions in hive ? Ex: I have a table landing hive> desc landing; OK id string iud string time timestamp content 1 I 2018-07-11 07:10:17 1 D 2018-07-11 07:11:16 2 I 2018-07-11 07:29:17 2 D 2018-07-11 07:31:21 3 I 2018-07-11 08:22:22 3 U 2018-07-11 09:12:22 5 I 2018-07-11 09:22:22 I want to insert into table B with following conditions -- When the 1) id has D flag then it should not be inserted (Skipped) , 2) when it has only U or I flag then insert , 3) when id has I,U followed by D flag then it should not insert (Skip) Output 3 U 2018-07-11 09:12:22 5 I 2018-07-11 09:22:22 I have written a code which is using row_number to eliminate duplicate records , but not satisfying the above conditions. select * from (select *,row_number() over(partition by id order by time desc) AS rowid FROM landing where iud <> 'D') r where r.rowid=1; 1 I 2018-07-11 07:10:17 1 2 I 2018-07-11 07:29:17 1 3 U 2018-07-11 09:12:22 1 5 I 2018-07-11 09:22:22 1 Help is appreciated . ... View more

Hi All, My table has 3776 rows . 1) when i do : explain select * from table Statistics: Num rows: 274 Data size: 110912 Basic stats: COMPLETE Column stats: NONE 2) when i do : explain select count(*) from table; Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE 3) when i do : explain select partner_id from table: Statistics: Num rows: 27728 Data size: 110912 Basic stats: COMPLETE Column stats: NONE Appreciate your help ... View more

Hi , Its a automated job (running through talend ) and its a common table where we will be performing insertions , sometimes it works fine and sometimes we get org.apache.hadoop.hive.ql.exec.MoveTask We are not able to figure it out the exact reason . Please help us . Table structure : CREATE EXTERNAL TABLE `lz_control_zyme_file_load_tracker`( `target_stage_name` string, `input_file_sequential_number` int, `source_system_file_type_name` string, `region_code` string, `source_file_name` string, `process_date` string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://HDPPRODNN/sales/channel/loading/lz_control_zyme_file_load_tracker/data' TBLPROPERTIES ( 'COLUMN_STATS_ACCURATE'='true', 'numFiles'='43739', 'numRows'='4706', 'rawDataSize'='443398', 'totalSize'='1842648', 'transient_lastDdlTime'='1520496190') ... View more