Support Questions
Find answers, ask questions, and share your expertise

Minimal executable jar based on Scala code packed with Maven

Solved Go to solution
Highlighted

Re: Minimal executable jar based on Scala code packed with Maven

Contributor

Hi @Vinod Bonthu,

your answer really helped but unfortunately it wasn't the complete solution. As statd in my error log I had to add some jars via the

spark-submit --jars

command and I also added my hive site with

spark-submit --files /usr/hdp/current/spark-client/conf/hive-site.xml

With those two changes and removing

.setMaster("local[2]")

it worked!

Thanks for your help!

Re: Minimal executable jar based on Scala code packed with Maven

Contributor

awesome @Ken Jiiii

hive-site.xml should be available across the cluster in /etc/spark/conf ( where /usr/hdp/current/spark-client/conf will be symlink to) and spark client need to be installed across the cluster worker nodes for your yarn-cluster mode to run as your spark driver can run on any worker node and should be having client installed with spark/conf. If you are using Ambari it will taking care of hive-site.xml available in /spark-client/conf/