Created 11-09-2022 08:38 PM
I am trying to create several tables with hundreds of millions of columns.
Tables are created and data is added while changing the orc.stripe.size value (64MB or 256MB) for each table because the size of each table is different. (If there is a lot of table data, an error occurred when setting 64MB, so the value was increased to 256MB.)
However, if the orc.stripe.size value is set large (ex. 256MB) for a table with a relatively small number of data, the following error occurs.
I don't know how to set this value and create it. Is there a way to set the value according to the table size?
Created 11-10-2022 05:45 AM
@octor I see you have already raised https://issues.apache.org/jira/browse/HIVE-26720
May I know why do you want to change stripe size?
Created 11-10-2022 05:46 PM
If the amount of table data is large (more than billions of rows), another error occurred when creating the table with the default stripe size of 64MB. So, when I increased it to 256MB, it was confirmed that it works normally.
Created 11-11-2022 12:40 AM
@octor You can tewk the split sizes if you are using Tez.
Created 11-11-2022 01:13 AM
The current tez settings are as follows.
- tez.grouping.max-size=52428800; -- 50MB
- tez.grouping.max-size=1073741824; -- 1GB
I think the max-size you mentioned is 64M.
set tez.grouping.min-size=16777216;--16 MB min split
set tez.grouping.max-size=64000000;--64 GB max split -> 64MB
From my experience while creating and adding data into tables, I think that if orc.stripe.size is set between 64MB and 256MB, i can create tables and add data smoothly. Can you tell me roughly what range I should take tez.grouping.min/max-size?
Created 11-11-2022 03:18 AM
You can set tez.grouping.max-size to 1gb.
Please increase below
set tez.runtime.io.sort.mb=4096; (40% of container size)