Member since
09-17-2018
8
Posts
0
Kudos Received
0
Solutions
09-06-2019
07:06 AM
Yes, we had the same dilemma when creating a fall-back queue as it doesn’t respect our model either! We observed the compress files in HDFS Oozie job to be allocated 1 container with 2GiB of memory and 1 VCore in YARN. We use a 5 VCore and 10GiB resource queue and the largest amount of data we’ve compressed is 100GiB. The YARN resource allocation doesn’t seem to change based on amount of data being compressed and therefore I think the YARN queue will not be limiting. As discussed earlier in the thread the architecture of the compress files in HDFS feature doesn’t appear to be very scalable: 1. All the data being compressed is first localized (copied) to a YARN Node Manager’s local cache (one directory is chosen from yarn.nodemanager.local-dirs). This requires enough local disk space on the partition where the directory resides. 2. The zip shell command is run locally on the same YARN node and uses 1x CPU core; the default zip compress is quite slow. 3. Enough space is required in local /tmp to hold a copy of the completed zip file before it is copied up to HDFS. Without any documentation on the compress files in HDFS feature this is just my opinion based on observations in our environment and reverse engineering. Kind regards, Julian
... View more
08-14-2019
08:50 PM
1 Kudo
Hi, Yes, they are. The hbase.quota.enabled property is not displayed in CDH. It must be added via a safety valve snippet under "Hbase Service Advanced Configuration Snippet (Safety Valve) for hbase-site.xml" in the HBase Configuration tab. Name: hbase.quota.enabled Value: true Description: Enable hbase quotas Also note that to then deploy the change requires a restart of multiple services, such as Impala, CDSW etc EDIT: I'm using CDH 6.2 Evan
... View more
01-18-2019
06:46 AM
HI, It's not uncommon for our documentaion team to cross-link pages to avoid duplication within our documentation. However with that said I'll pass your feedback along to our documentation team.
... View more