Member since
05-12-2021
5
Posts
0
Kudos Received
0
Solutions
09-01-2021
10:28 PM
1 Kudo
Hi @saikat As I can understand you are running a merge query and it is failing with java.lang.OutOfMemoryError error. Step 1: Could you please run major compaction on all the tables involves in the merge query(If it is an ACID table or else ignore step1). Once the major compaction is triggered make sure it got completed by running "show compactions;" command in the beeline. This will bring down some stats collection burden for the hive. How to run minor and major compaction? Alter table <table name> compact 'MAJOR'; Step 2: Once step1 is done. Please set the following propery in beeline session level and re-run the merge query set hive.tez.container.size=16384; set hive.tez.java.opts=-Xmx13107m; set tez.runtime.io.sort.mb=4096; set tez.task.resource.memory.mb=16384; set tez.am.resource.memory.mb=16384; set tez.am.launch.cmd-opts=-Xmx13107m; set hive.auto.convert.join=false; The TEZ container and AM size is set as 16GB, if the query got failed you can increase the value to 20GB(then hive.tez.java.opts and tez.am.launch.cmd-opts need to be configured 80% of container and AM size that is 16384). If the query got succeeded with 16GB of TEZ container and AM size then you can try to decrease it too 14/12/10 and figure out a benchmark where it is failing and getting succeeded. In this way, you can save resources. If you are happy with the comment, Mark it "Accept as Solution".
... View more
05-25-2021
10:43 AM
Hello, Hive compaction runs first, then a cleaner thread waits for all readers to finish reading the old base/delta files. When it determines that nobody is reading the old files anymore, only then does the deletion of the old files occur. Please run SHOW COMPACTIONS and look at the state of the compaction in question. Also would help to know which version are you on for CDH/HDP/CDP? If CDP, public or private cloud? Hope this helps, Alex
... View more
05-15-2021
10:29 PM
@Daming Xue Namespace is created. [serverUri=node6.cloudera.com:10000;version=3.1.3000.7.1.3.0-100;sequence=0000000018]
... View more