How do I view cluster specifications?

[#1]

I'm very new to Cloudera and Spark. I need to know the specifications of the cluster I'm using, so I can better optimize my Spark session configuration (executors/cores/ram/etc). I'm trying to view these details in Cloudera Manager, but I'm not sure where to look. I want to know on the specific cluster I'm using, how many nodes it has, and how many cores and how much memory per node? Forgive my naivety, but some guidance would be appreciated!

[#2 · Accepted solution]

Hello @Data1701 ,

The metrics you are looking for would be under the Hosts view in Cloudera Manager. On the left-hand panel in Cloudera Manager click on Hosts > All Hosts. This will list all of the hosts in the cluster, along with their physical memory and cores for each. Note that, by default, not all of the necessary columns may be visible in the table. To add columns of interest (e.g. cores), in the top right corner next to "Columns:" click Selected dropdown menu and choose the columns you need to see.

Hope this helps.

Regards,

Alex

[#3]

I did need to manually add 'cores' to the list, but I got it now. Thanks!

[#4]

Happy to hear, @Data1701 . Accept reply as Solution if the answer helped.