Yarn does not do this. There is no relationship AFAIK. It is done by the oozie server. If oozie doesn't do it ( which can happen ) the jobs run on till they finish. ... View more

Hi Klaus, Hadoop on 1GB doesn't make much sense. Also YARN is not really flexible when it comes to memory allocation. The nodemanagers get a maximum of memory they can use to schedule yarn tasks ( mapreduce, spark, ... ) yarn.nodemanager.resource.memory-mb And a minimum which is also the multiple of task slots available yarn.scheduler.minimum-allocation-mb So if you have a maximum of 16GB and a minimum of 1GB he can give out up to 16 task slots to applications. But AFAIK there is no way to dynamically change that without restarting the nodemanagers. Yarn makes sure no task exceeds the slots it is provided but it doesn't give a damn about Operation System limits. So if you give your VM 4GB and set the yarn memory to 32GB yarn will happily schedule tasks till your system goes to its knees. You can of course enable swapping but that will result in bad performance. So summary: Flexible memory settings on a nodemanager node is not a good idea. ... View more

I assume you mean a map-side join in Hive. ( I.e. small dataset is replicated to all map tasks and then join is done on map side vs. the standard shuffle or distributed join which distributes both tables around. ) Its actually easy. Assume you have 1 table with 1TB and 1 table with 1MB Assume as well that we have 50 nodes. ( I think he only copies datasets for the distributed cache once per node and not once per task might be wrong ) So MapSide Join: You have to copy the small table to every node: 50x1MB = 50MB of data are copied across the network Shuffle Join: Both tables need to be copied once i.e. 1TB + 1MB = 1.00001 TB will be copied over the network. MapSide Join is much better than Shuffle join in this case. Assume you have 1 table with 1TB and one table with 500GB: MapSide Join: 500GB table needs to be copied to 50 nodes for 50 * 500 = 25 TB of data being copied over the network Shuffle Join: 1.5TB of data need to be copied over the network. So in this case MapSide join is much worse than shuffle join. And the CBO tries to figure out which one is better. This gets harder because of where conditions etc. The same is true for join types in pig/Spark etc. You can do the math yourself 🙂 ... View more

Depends what you are trying to do. - One thing buckets are used for is to increase load performance Essentially when you load data you often do not want one load per mapper ( especially for partitioned loads because this results in small files ), buckets are a good way to define the number of reducers running. So if your cluster has 40 task slots and you want the fastest ORC creation performance possible you would want 40 buckets. ( Or distribute your data by 40 and set number of reducers ) http://www.slideshare.net/BenjaminLeonhardi/hive-loading-data - SELECT performance ( predicate pushdown ) Buckets can help with the predicate pushdown since every value belonging to one value will end up in one bucket. So if you bucket by 31 days and filter for one day Hive will be able to more or less disregard 30 buckets. Obviously this doesn't need to be good since you often WANT parallel execution like aggregations. So it depends on your query if this is good. It might be better to sort by day and bucket by something like customer id if you have to have buckets for some of the other reasons. - Join Performance ( bucket join ) Buckets can lead to efficient joins if both joined tables are bucketed on the join key since he only needs to join bucket with bucket. This was big in the old times but is not that applicable anymore with cost based optimization in newere Hive versions ( since the optimizer already is very good at choosing mapside vs shuffle join and a bucket join can actually stop him from using the better one. - Sampling performance Some sample operations can get faster with buckets. So to summarize buckets are a bit of an older concept and I wouldn't use them unless I have a clear case for it. The join argument is not that applicable anymore, the increased load performance also is not always relevant since you normally load single partitions where a map only load is often best. Select pushdown can be enhanced but also hindered depending how you do it and a SORT by is normally better during load ( see document ). And I think sampling is a bit niche. So all in all avoid if you don't know too well what you are doing, and regarding size? 1-10 blocks is a pretty good size for hadoop files. Buckets should not be much smaller ( unless you want very fast answer rates on small amounts of data no rule without exceptions ) ... View more