Perhaps this has already been posted elsewhere. If so you have my apology, but I was trying to install HDP 2.6 on a Centos 7 everything distribution and ran into an issue where the Spark client failed to install correctly. I discovered that the link at /usr/hdp/current/spark-client/conf for the Spark configuration was not pointing to an existent directory. So as root I created the directory with:
mkdir -p /etc/spark/conf
and reran the install wizard and all was well.
-=> Gregg <=-
It also solved my problem with spark2.
I experienced same problem with spark2, and I did what Gregg suggested same for spark2:
I run below command the servers which I cannot accomplish the installation of spark2 via Ambari.
mkdir -p /etc/spark2/conf
And result is succesfull.
Thanks again @Gregg Leichtman.