spark-submit --master yarn --deploy-mode cluster sqlserver.py --jars sqljdbc42.jar
I get Error : java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver
Everything works when i use --deploy-mode client and copy the jar file to /usr/hdp/current/spark2-client/jars/sqljdbc42.jar
Should i copy the sqljdbc42.jar to /usr/hdp/current/hadoop-yarn-client/lib/ on all data nodes?
Spark driver is not able to find the sqljdbc.jar in class path.
spark-submit, the application jar along with any jars included with the
--jars option will be automatically transferred to the cluster. URLs supplied after
--jars must be separated by commas. That list is included in the driver and executor classpaths. Directory expansion does not work with
1) Provide the spark.driver.extraClassPath =/usr/hdp/hive/lib/mysql-connector-java.jar
2) Provide the spark.executor.extraClassPath = /usr/hdp/hive/lib/mysql-connector-java.jar.
3) Add Sqljdbc.jar to the Spark Classpath or add it using -jar option.
Hope this helps you.
Thanks Sridhar Already tried the --jars and spark.driver.extraClassPath & spark.executor.extraClassPath, they don't work. What do you mean by "Directory expansion does not work with --jars"
Try this only replace the path on jars and make sure sqlserver.py on working directory (rest leave it as is)
spark-submit --master yarn --deploy-mode cluster --jars /path/to/driver/sqljdbc42.jar --conf "spark.driver.extraClassPath=sqljdbc42.jar" --conf "spark.executor.extraClassPath=sqljdbc42.jar" sqlserver.py
*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.
None of the above worked. It working after adding below configuration