Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark version 2.4.0, Issues with the microsoft JDBC driver mssql-jdbc-7.4.1.jre8.jar, error as No Suitable Driver or ClassNotfoundException

Highlighted

Spark version 2.4.0, Issues with the microsoft JDBC driver mssql-jdbc-7.4.1.jre8.jar, error as No Suitable Driver or ClassNotfoundException

New Contributor
 
1 REPLY 1

Re: Spark version 2.4.0, Issues with the microsoft JDBC driver mssql-jdbc-7.4.1.jre8.jar, error as No Suitable Driver or ClassNotfoundException

New Contributor

I create the pyspark code and ran it on my local windows machine and everything seems to be running fine. While moving to hadoop server, spark2-submit command is not able to pick up the driver for MSSQL server. Following are the steps done to make it to work. 1) Moved the MSSQL jar file (mssql-jdbc-7.4.1.jre8.jar) to the spark home under the jars folder. 2) Added -- jar "pathtothejdcjarfile" using the local file system file://pathto jar file and jars folder in step 1. 3) used the classpath module to set the path for the jar file as below --conf "spark.driver.extraClassPath=pathtojdbcjarfile" --conf "spark.executor.extraClassPath=pathtoJDCjarfile" 4) Final spark submit command. spark2-submit --jar "pathtoJDBCjarfile" --conf "spark.driver.extraClassPath=pathtojdbcjarfile" --conf "spark.executor.extraClassPath=pathtoJDCjarfile" --master yarn --deploy-mode client --executor-memory 1g /home/aab9010/SOMEECRIPT.py

All the efforts leads to the same error either Class not found or No suitable driver.

Any help is appreciated.

Don't have an account?
Coming from Hortonworks? Activate your account here