CDP Public Cloud DL /DH vertical scaling for Azure based cluster

[#1]

Why scale vertically?

Once a DL or DH is provisioned with given parameters (instance types, root volume size etc), later there is currently no way in CDP / cloudbreak to change those parameters. However, with time the load may increase and it may not be possible to terminate and re-launch the DL or DH. In this case you might need to scale vertically via Azure console.

On the other hand this scaling plays only partially with CDP features, so the long-term fix is rather upgrading to a larger duty size.

Overview

Both VM and root disk resize steps do require the VM to be in a stopped state. So you can stop the DH or DL and perform both steps.

Scaling the virtual machine size

Based on the azure doc for resizing VMs.

After resizing the VM the new size will be in effect until you do a repair or upgrade. After that you will need to do the resize again, if you do not have HA setup.

If, however, you have a HA setup, keep in mind to change both VMs: in that way, when the repair changes the primary host to the secondary, latter will have the correct size already.

1. The VM has to be in a stopped state. If the VM is in an availability set, then all VMs in that availability set have to be stopped. In case of cloudbreak, that means a whole hostgroup. That is, if you have a datahub, then stop the datahub, if you have a datalake then you will need to stop the environment.

2. Go to “Azure portal” → “Virtual Machines” and find your machines. Click on the one you want to resize to go into the details page of the vm. There, look for submenu Size, and click on it.

3. In the next menu you will get a list of available VM sizes. Please note, that you cannot resize from any type of VM to any other type. It’s best to stay within the same series, i.e. if you have a Standard_D8s v3 VM, then select a bigger one within the Ds v3 series. Click on the desired size and click resize at the bottom of the page.

Changing the root disk size

After changing the root disk size the new value will be in effect until you do a repair or upgrade. After that you will need to do the resize again, if you do not have HA setup.

If, however, you have a HA setup, keep in mind to change root disk for both VMs: in that way, when the repair changes the primary host to the secondary, latter will already have the larger root disk size.

1. Please make sure that the VM is in a stopped state. In case of a DH please stop the DH, in case of a DL you need to stop the environment.

2. Go to azure portal, and select the VM where you want to resize the root disk. On the details page click on “Disks”. Disks will be listed in two groups, the upper one will be the root disk. Click on it.

3. On the upcoming details page please click on “Size and performance“. You will get a list of available root disk SKUs and size. In the custom disk size text box please enter the desired disk size. Note, that this may change the SKU for the disk. On the bottom of the page please press resize.

4. Going back to the VM “Disks” menu you should see the new value after a refresh. However, on the VM, you will need to resize the partition and the filesystem manually.

5. Start the DH or environment, and after the VM started ssh into it. Issue two commands: lsblk will show you the size of the attached disk and the size of the partition mounted from that disk.
   Disk sda is reported to have 567G (the size we have set in azure portal), however sda2 is still 133.5G, the original size.

6. Issue command growpart /dev/sda 2 to grow the partition that is mounted under /

   1[root@gpapp-azvolres5-dl-master0 cloudbreak]# growpart /dev/sda 2 2CHANGED: partition=2 start=1026048 old: size=279992287 end=281018335 new: size=1188059103 end=1189085151

   If you now again issue lsblk then you will see that the partition was grown

7. You thus now need to resize the filesystem under / , issuethe command xfs_growfs / to do the resize.

Resizing other disks than the root disk

From cloudbreak point of view there are 3 types of disks on an azure linux VM:

The root disk: can be resized as discussed above, but repair / upgrade reverts the change

• A temporary storage: size is fixed, depends on VM size.

• Data disks: can be resized, but might not make much sense.