Hello, I have a table (not partitioned), with tons of small files... So, I had run an "alter table concat" and it works, but not in the way i thought... From tons of small files (let say 10 000), i get 1 000 ; i run again and i get 100, to converge to 5 files... I use a lot of SET instructions but I think I used them without understanding their purpose. So, is there a way to concat a table in a only file, in a one shot ? Thanks 🙂 SF ... View more

Hi, I am a little bit confused about the FIFO execution mode and its behaviour. For example, a coordinator "C" generates 4 instances of a Worfklow at those nominal times : 8:00am, 7:45am, 7:30am, 7:15am For example, considering they are waiting for resources (a flag file for example) until 8:07am (the flag is available), I note that the 8:00am instance is executed first. And when it is done, 7:15 is executed, then 7:30 then 7:45. Isn't it weird ? Did I miss something ? Thanks 🙂 ... View more

Hi, I read in many articles about Hive queries optimization the advice that consists in presort tables in order to optimize joins. I read too, in others articles, that the sort algorithm (when shuffling) used is the QuickSort or derived... So, I am a little bit confused, is the Quick Sort fastest when it takes an array composed of 2 sorted arrays ? Thanks. ... View more

Hi, For example, I have an Hive query which implies a Map phase and a Reduce phase. Is there a way to get the output file from the Map phase, before it is processed by the Reduce phase ? That will allow me to understand who does what (and then, optimize the query)... Thanks. ... View more

Hi, I'm a little bit confused with YARN fair scheduler, I'll be glad somebody helps me :-). Let's say that I run a TEZ application (an HIVE query for example implying a simple Map/Reduce) with the following parameters : yarn.scheduler.maximum-allocation-vcores=16 yarn.scheduler.minimum-allocation-mb=2048 mapreduce.map.memory.mb=4096 mmapreduce.reduce.memory.mb=8192 When I run my query, it will demand to the Resources Manager : - an Application Master creation (including the DAG) (??? GB) - a node manager creation for the Map task (2x2GB) - a node manager creation for the Reduce task (4x2GB) 1) If the cluster is busy (not enought memory available), with the FIFO scheduler, my query will be placed in queue, waiting for the freeing of resources (1 container for the AM, 1 container of 4GB for the map task and 1 container of 8GB for the reduce task). First of all, am I right ? 2) With the fair scheduler, I read "With the Fair Scheduler, there is no need to reserve a set amount of capacity, since it will dynamically balance resources between all running jobs." That will say that I will not wait to get the 3 enought sized containers and then, that I will get (for example) a container of 2GB for the Map task and a container of 2GB for the reduce task IF the average container size is 2GB ? In this case, increasing mapreduce.map|reduce.memory.mb is not useful ? Is that so ? Thanks a lot to help me to see clearer 🙂 ... View more

Hi, I noticed something curious about datasets (used in output-events). Considering a coordinator "A" (the dependence) with these parameters : frequency="${coord:minutes(5)} dataset / frequency : ${coord:hours(1)} dataset / uri-template : /temp/${YEAR}/${MONTH}/${DAY}/${HOUR} Considering a coordinator "Z" waiting for the dependencies and executing itself every 5 minutes, started at 2:18 for example... I have this when I look the coordinator "Z": nominal time | missing dependencies |last modified time -------------+-----------------------+------------------ 8:10 | /temp/2017/11/08/07 | 7:18 8:15 | /temp/2017/11/08/07 | 7:18 8:20 | /temp/2017/11/08/08 | 8:18 What I understand is : The modification time implies the dependency evaluation Before 8:18, it is still the 2017/11/08/07 file to be needed, because the previous modification is at 7 (7:18 precisely) It is the normal behaviour ? Or am I doing something weird somewhere 🙂 ? Thanks. ... View more

Hi, In my organization, Hive is used with the hive.support.concurrency setted to false. I am wondering what are the consequences about inserting datas during a select (and vice versa). At insert, I think the table's metadatas are updated at the very end of the Map/Reduce job. Thus, a select should be not disturbed, because I think files involved by the select are determined at the very beginning of the M/R job... For an insert overwrite, I think this is pretty similar, but I didn't find a confirmation during my research... Could you validate (or not ;)) my thoughts ? Thanks 🙂 ... View more

Hi, I would like to know if there is a way to get metrics about cpu and memory usage. For example, I would like to highlight the effect of a skew join on reducers, disk I/O, mapper memory usage during the querying, etc. I saw really interesting slides about performance comparison (Hortonworks), with some graphes and bars, etc. and I was wondering the method to get those values... Thanks. ... View more