I had installed my big data stack using Ambari and was running it successfully for some basic stuff. But now getting new issues while running with yarn/phoenix. After some juggernaut I found that I have two spark configuration directories(even there are difference directories for other big data stack modules too) named
/usr/hdp/126.96.36.199-258/etc/spark/conf and another is at
I found that spark-env.sh is different in both these directories. May I know why these are different and what is significance of these directories ? Configuration present in
/usr/hdp/188.8.131.52-258/spark/conf is what I see in my Ambari GUI.
Can you check o/p of following command :
$ls -al /usr/hdp/184.108.40.206-258/spark
To me, '/usr/hdp/220.127.116.11-258/etc/spark/conf' doesn't look correct. Was this symlink (most probably) manually done anytime ?
/usr/hdp/18.104.22.168-258/spark/conf is the correct path.
O/p on my cluster, conf is a symlink :
# ls -al /usr/hdp/22.214.171.124-1035/spark
lrwxrwxrwx 1 root root 25 Jul 21 21:57 conf -> /etc/spark/126.96.36.199-1035/0
No I did't do anything manually. But you are right in assuming that conf at
/usr/hdp/188.8.131.52-258/spark is a symlink.
conf -> /etc/spark/184.108.40.206-258/0
But It seems the other directory(/usr/hdp/220.127.116.11-258/etc/spark/conf') is in use and my guess is that it is for other nodes side of spark.I think some one from hortonworks can clarify its use ?