Yes, we had the same dilemma when creating a fall-back queue as it doesn’t respect our model either! We observed the compress files in HDFS Oozie job to be allocated 1 container with 2GiB of memory and 1 VCore in YARN. We use a 5 VCore and 10GiB resource queue and the largest amount of data we’ve compressed is 100GiB. The YARN resource allocation doesn’t seem to change based on amount of data being compressed and therefore I think the YARN queue will not be limiting. As discussed earlier in the thread the architecture of the compress files in HDFS feature doesn’t appear to be very scalable: 1. All the data being compressed is first localized (copied) to a YARN Node Manager’s local cache (one directory is chosen from yarn.nodemanager.local-dirs). This requires enough local disk space on the partition where the directory resides. 2. The zip shell command is run locally on the same YARN node and uses 1x CPU core; the default zip compress is quite slow. 3. Enough space is required in local /tmp to hold a copy of the completed zip file before it is copied up to HDFS. Without any documentation on the compress files in HDFS feature this is just my opinion based on observations in our environment and reverse engineering. Kind regards, Julian
... View more
