spark-submit class not found

[#1]

spark-submit --master yarn --deploy-mode cluster sqlserver.py --jars sqljdbc42.jar

I get Error : java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver

Everything works when i use --deploy-mode client and copy the jar file to /usr/hdp/current/spark2-client/jars/sqljdbc42.jar

Should i copy the sqljdbc42.jar to /usr/hdp/current/hadoop-yarn-client/lib/ on all data nodes?

[#2]

@vishal dutt

Spark driver is not able to find the sqljdbc.jar in class path.

When using spark-submit, the application jar along with any jars included with the --jars option will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. That list is included in the driver and executor classpaths. Directory expansion does not work with --jars.

Else

1) Provide the spark.driver.extraClassPath =/usr/hdp/hive/lib/mysql-connector-java.jar
2) Provide the spark.executor.extraClassPath = /usr/hdp/hive/lib/mysql-connector-java.jar.

3) Add Sqljdbc.jar to the Spark Classpath or add it using -jar option.

Hope this helps you.

[#3]

Thanks Sridhar Already tried the --jars and spark.driver.extraClassPath & spark.executor.extraClassPath, they don't work. What do you mean by "Directory expansion does not work with --jars"

[#4]

@vishal dutt

Try this only replace the path on jars and make sure sqlserver.py on working directory (rest leave it as is)

spark-submit --master yarn --deploy-mode cluster --jars /path/to/driver/sqljdbc42.jar --conf "spark.driver.extraClassPath=sqljdbc42.jar" --conf "spark.executor.extraClassPath=sqljdbc42.jar" sqlserver.py

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#5]

None of the above worked. It working after adding below configuration

set("spark.jars","/usr/hdp/current/spark2-client/jars/sqljdbc42.jar")