Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Set SPARK_HOME for spark2

Solved Go to solution

Set SPARK_HOME for spark2

Explorer

Spark2 2.3.0 on cdh.5.13 deployed with CSD and parcels.

 

by default it uses /etc/spark2/conf,

but it seems it should use /etc/spark2/conf.cloudera.spark2_on_yarn/

 

is there a way to set it properly?

i could set it in spark2-env.sh but it seems not the appropriate place.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Set SPARK_HOME for spark2

Guru
@sbn,

/etc/spark2/conf should be a symlink to /etc/spark2/conf.cloudera.spark2_on_yarn, can you confirm by running:

ls -al /etc/spark2

Cheers
Eric
2 REPLIES 2
Highlighted

Re: Set SPARK_HOME for spark2

Guru
@sbn,

/etc/spark2/conf should be a symlink to /etc/spark2/conf.cloudera.spark2_on_yarn, can you confirm by running:

ls -al /etc/spark2

Cheers
Eric

Re: Set SPARK_HOME for spark2

Explorer

thanks, i suppose one could just move that to wherever you want the home to be.

this was part of a hack deployment, so ultimately not needed.

Don't have an account?
Coming from Hortonworks? Activate your account here