question Re: CDP Public Cloud - Resizing of Worker/Compute Nodes in Support Questions

CDP Public Cloud - Resizing of Worker/Compute Nodes

AndreaCavenago — Fri, 27 Oct 2023 10:55:24 GMT

Hello,

due to a lack of memory on a DE cluster in DataHub while executing Spark jobs, we are evaluating the resizing of worker/compute nodes with more performant EC2 instance (no need to add more nodes, just increase the available memory).

At the moment we have 3 worker nodes and 4 compute nodes, running on m5.4xlarge instances.

Is it better to resize both the nodes type or we can resize, for example, only the compute nodes?

Thank you,

Andrea

Re: CDP Public Cloud - Resizing of Worker/Compute Nodes

aakulov — Fri, 27 Oct 2023 23:34:42 GMT

Hi @AndreaCavenago ,

In a Data Engineering Data Hub, the difference between Worker group and a Compute group is the presence of HDFS service on the Worker nodes (see here). For up-scaling the node type, I would recommend doing Compute group first as that is the group is meant to be ephemeral and come off and on-online dynamically. Do Compute group first and then evaluate if you really need to scale-up the Worker group next.

And remember that you can do the scale-up operation both from the UI as well as through CLI. And with the CLI you can provide the specific the --group <name> parameter (see here).

Hope that helps.

Kind regards,

Alex

Re: CDP Public Cloud - Resizing of Worker/Compute Nodes

DianaTorres — Thu, 02 Nov 2023 20:39:46 GMT

@AndreaCavenago Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Re: CDP Public Cloud - Resizing of Worker/Compute Nodes

AndreaCavenago — Thu, 02 Nov 2023 23:01:52 GMT

@DianaTorres, yes, thank you!

@aakulovthank you for your answer!

Andrea