Default Spark env config error in HDP2.4 installation

[#1]

Default Spark env config error in HDP2.4 installation:

#SPARK_DRIVER_MEMORY="512 Mb"

This line include a wrong config param "512 Mb", it should be "512M" without space. This will cause spark-shell yarn client start error:

Invalid initial heap size.

Suggest to update the template.

[#2 · Accepted solution]

@JR Cao

Hi, I think that line is commented out so won't cause any issue by default and YES if user uncommented that line then it will cause error. But I will also share this to spark team to change the default template.

Thanks for the feedback.

[#3]

@JR Cao

I have raised a internal HDP Spark jira BUG-57627 for this issue. Thanks for raising this.

[#4]

@JR Cao please accept my post as an answer if you are good with the provided information.

[#5]

@vshukla