Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark stanalone mode there is library issue ?

Highlighted

Spark stanalone mode there is library issue ?

Contributor

Hi All,

When I am running the spark in local mode my code is working fine and when I am running it in standalone mode and deploy mode as client , I am getting the class not found exception "Caused by: java.lang.ClassNotFoundException:" for my custom class .Actually I am distributing the jar which has the class to worker by setting the property in spark configuration sparkConfig.setJars(new String[]{"myjar.jar"}); and I am seeing this jar getting distributed on worker node as well ,but I am getting the class not found exception.Why the executor is not able to fine the class in the jar on the worker node.

Could someone please let me what mistake I am making here .

Thanks in advance ,

Param.

2 REPLIES 2

Re: Spark stanalone mode there is library issue ?

Guru

@Param NC, Can you please try to add the necessary jar to --driver-class-path and --jars option while running application?

https://support.datastax.com/hc/en-us/articles/207442243-Spark-submit-fails-with-class-not-found-whe...

Re: Spark stanalone mode there is library issue ?

Contributor

@yvora

Actually I am submitting my spark application through program , I have done the same thing programatically by adding the sparkConfig.setJars(new String[]{"myjar.jar"}); still it is not working .