Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Default Spark env config error in HDP2.4 installation

Solved Go to solution
Highlighted

Default Spark env config error in HDP2.4 installation

New Contributor

Default Spark env config error in HDP2.4 installation:

#SPARK_DRIVER_MEMORY="512 Mb"

This line include a wrong config param "512 Mb", it should be "512M" without space. This will cause spark-shell yarn client start error:

Invalid initial heap size.

Suggest to update the template.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Default Spark env config error in HDP2.4 installation

@JR Cao

Hi, I think that line is commented out so won't cause any issue by default and YES if user uncommented that line then it will cause error. But I will also share this to spark team to change the default template.

Thanks for the feedback.

View solution in original post

4 REPLIES 4
Highlighted

Re: Default Spark env config error in HDP2.4 installation

@JR Cao

Hi, I think that line is commented out so won't cause any issue by default and YES if user uncommented that line then it will cause error. But I will also share this to spark team to change the default template.

Thanks for the feedback.

View solution in original post

Highlighted

Re: Default Spark env config error in HDP2.4 installation

@JR Cao

I have raised a internal HDP Spark jira BUG-57627 for this issue. Thanks for raising this.

Highlighted

Re: Default Spark env config error in HDP2.4 installation

@JR Cao please accept my post as an answer if you are good with the provided information.

Highlighted

Re: Default Spark env config error in HDP2.4 installation

Mentor
Don't have an account?
Coming from Hortonworks? Activate your account here