https://community.cloudera.com/t5/Support-Questions/Spark-submit-to-deploy-the-jar-to-Spark-java-lang/m-p/197809#M159860 <P>Hi, I'm doing the following tutorial </P><P><A href="https://es.hortonworks.com/tutorial/deploying-machine-learning-models-using-spark-structured-streaming/#deploying-the-model"></A><A href="https://es.hortonworks.com/tutorial/deploying-machine-learning-models-using-spark-structured-streaming/#deploying-the-model" target="_blank">https://es.hortonworks.com/tutorial/deploying-machine-learning-models-using-spark-structured-streaming/#deploying-the-model</A> <BR /></P><P>I have reached this point of the tutorial: Then use spark-submit to deploy the jar to Spark.</P><P>I am trying to submit a job which is in target/main/scala, which is a jar file, with the following lines:</P><PRE>/usr/hdp/current/spark2-client/bin/spark-submit --class "main.scala.Collect" --master local[4] ./SentimentAnalysis-assembly-2.0.0.jar</PRE><P>All goes well, except the following errors:</P><PRE>SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See <A href="http://www.slf4j.org/codes.html#StaticLoggerBinder" target="_blank">http://www.slf4j.org/codes.html#StaticLoggerBinder</A> for further details. root |-- value: string (nullable = true) Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/streaming/Source$class at org.apache.spark.sql.kafka010.KafkaSource.&lt;init&gt;(KafkaSource.scala:84) at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:152) at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:240) at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$1.applyOrElse(StreamingQueryManager.scala:245) at org.apache.spark.sql.streaming.StreamingQueryManager$$anonfun$1.applyOrElse(StreamingQueryManager.scala:241) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:279) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:279) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:278) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:321) at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179) at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:319) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:321) at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179) at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:319) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:321) at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179) at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:319) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$1.apply(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:321) at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:179) at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:319) at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:284) at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:268) at org.apache.spark.sql.streaming.StreamingQueryManager.startQuery(StreamingQueryManager.scala:241) at org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:287) at main.scala.Collect$.main(Collect.scala:90) at main.scala.Collect.main(Collect.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.Source$class at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 47 more</PRE><P>my build.sbt file is:</P><PRE>name := "SentimentAnalysis" version := "2.0.0" scalaVersion := "2.10.5"//"2.10.4"// libraryDependencies ++= { val sparkVer = "2.1.0"//"1.6.1"// Seq( "org.apache.spark" %% "spark-core" % sparkVer % "provided" withSources(), "org.apache.spark" %% "spark-mllib" % sparkVer % "provided" withSources(), "org.apache.spark" %% "spark-sql" % sparkVer % "provided" withSources(), "org.apache.spark" %% "spark-streaming" % sparkVer % "provided" withSources(), "org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVer withSources(), "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVer withSources(), "org.apache.kafka" %% "kafka" % "0.10.0" withSources(), "com.typesafe" % "config" % "1.3.1", "com.google.code.gson" % "gson" % "2.8.0" ) } assemblyMergeStrategy in assembly := { case PathList("org", "apache", xs @ _*) =&gt; MergeStrategy.first case PathList("javax", "xml", xs @ _*) =&gt; MergeStrategy.first case PathList("com", "esotericsoftware", xs @ _*) =&gt; MergeStrategy.first case PathList("com", "google", xs @ _*) =&gt; MergeStrategy.first case x =&gt; val oldStrategy = (assemblyMergeStrategy in assembly).value oldStrategy(x) }</PRE><P>If anyone can help me thank you very much.</P><P><A href="https://es.hortonworks.com/tutorial/deploying-machine-learning-models-using-spark-structured-streaming/#deploying-the-model"></A> </P> Wed, 16 May 2018 22:10:33 GMT dsandoval 2018-05-16T22:10:33Z