difference in config of /usr/hdp/2.4.2.0-258/etc/spark/conf and /usr/hdp/2.4.2.0-258/spark/conf

[#1]

I had installed my big data stack using Ambari and was running it successfully for some basic stuff. But now getting new issues while running with yarn/phoenix. After some juggernaut I found that I have two spark configuration directories(even there are difference directories for other big data stack modules too) named

/usr/hdp/2.4.2.0-258/etc/spark/conf and another is at

/usr/hdp/2.4.2.0-258/spark/conf

I found that spark-env.sh is different in both these directories. May I know why these are different and what is significance of these directories ? Configuration present in

/usr/hdp/2.4.2.0-258/spark/conf is what I see in my Ambari GUI.

[#2]

Can you check o/p of following command :

$ls -al /usr/hdp/2.4.2.0-258/spark

To me, '/usr/hdp/2.4.2.0-258/etc/spark/conf' doesn't look correct. Was this symlink (most probably) manually done anytime ?

/usr/hdp/2.4.2.0-258/spark/conf is the correct path.

O/p on my cluster, conf is a symlink :

# ls -al /usr/hdp/2.5.0.0-1035/spark

....

lrwxrwxrwx 1 root root 25 Jul 21 21:57 conf -> /etc/spark/2.5.0.0-1035/0

....

[#3]

No I did't do anything manually. But you are right in assuming that conf at

/usr/hdp/2.4.2.0-258/spark is a symlink.

conf -> /etc/spark/2.4.2.0-258/0

But It seems the other directory(/usr/hdp/2.4.2.0-258/etc/spark/conf') is in use and my guess is that it is for other nodes side of spark.I think some one from hortonworks can clarify its use ?