04-03-2017 09:21 AM
No, I meant the gateway role or if needed spark history server role where you don't see spark-default.conf having a correct config.
Gateway role is better. If you don't want to delete. Just add a new gateway role from CM->Spark->Instances and then see if then configuration comes correct.
04-03-2017 11:03 AM
1- removed all instances from spark gateway, deployed configuration and then added the servers to the gatway and deployed the configuration but this didn't help.
2- checked all the luster instances and see the new spark history server in all spark-default.conf and no servers has any other configuration.
cat /etc/spark/conf/spark-defaults.conf | grep spark.yarn.historyServer.address
3- the valve CM config is empty.
Where these conf can be overwritten in some other places?