CDP Public Cloud - Resizing of Worker/Compute Nodes

[#1]

Hello,

due to a lack of memory on a DE cluster in DataHub while executing Spark jobs, we are evaluating the resizing of worker/compute nodes with more performant EC2 instance (no need to add more nodes, just increase the available memory).

At the moment we have 3 worker nodes and 4 compute nodes, running on m5.4xlarge instances.

Is it better to resize both the nodes type or we can resize, for example, only the compute nodes?

Thank you,

Andrea

[#2 · Accepted solution]

Hi @AndreaCavenago ,

In a Data Engineering Data Hub, the difference between Worker group and a Compute group is the presence of HDFS service on the Worker nodes (see here). For up-scaling the node type, I would recommend doing Compute group first as that is the group is meant to be ephemeral and come off and on-online dynamically. Do Compute group first and then evaluate if you really need to scale-up the Worker group next. 

And remember that you can do the scale-up operation both from the UI as well as through CLI. And with the CLI you can provide the specific the --group <name> parameter (see here).

Hope that helps.

Kind regards,

Alex

[#3]

@AndreaCavenago Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

[#4]

@DianaTorres, yes, thank you!

@aakulovthank you for your answer!

Andrea