Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

IBM Power - Spark worker node sizing

IBM Power - Spark worker node sizing

New Contributor

Reaching out to the community for guidance or advice on sizing Spark worker nodes on IBM Power servers. Given the chip architecture has 8 threads per core compared to 2 threads per core for Intel x64, should I adjust memory? I don't want to over configure but also don't want constrain throughput by not having enough memory. Thanks in advance for input or guidance.

3 REPLIES 3
Highlighted

Re: IBM Power - Spark worker node sizing

New Contributor

Check out IBM Redbooks regarding SPARK here: http://www.redbooks.ibm.com/redbooks.nsf/searchsite?SearchView=&query=spark&SearchWV=true

Usually they have Sizing guidelines there.

Highlighted

Re: IBM Power - Spark worker node sizing

New Contributor

Thanks for the links to the Redbooks - I checked them (although admittedly did not read each one of them thoroughly) and wasn't able to get a definitive answer. On a separate note, this post by Raj Krishamurthy and Randy Swanberg has some good tips for tuning Spark on Power8.

Highlighted

Re: IBM Power - Spark worker node sizing

New Contributor

I didn't search all the Redbooks, but in "IBM Data Engine for Hadoop and Spark" you can find "Solution reference architecture" chapter and there is some information on sizing.
Also in the book "Hortonworks Data Platform with IBM Spectrum Scale Reference Guide for Building an Integrated Solution" they have the following: "For information and assistance about sizing and configuring the HDP on a Power Systems + IBM Spectrum Scale/IBM Elastic Storage Server solution, contact the Cognitive Systems Solution Center (cssc@us.ibm.com)." Also there is some information there about reference architecture they used.

Don't have an account?
Coming from Hortonworks? Activate your account here