Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error CLOUDERA - wordcounts

Highlighted

Error CLOUDERA - wordcounts

Contributor

Hi

I am new in Spark. I am working on Cloudera Standalone. I have downloaded Eclipse JDK 8 on my local and written a wordcount program in Scala :-

 

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
 
object WordCount extends App {
  val conf = new SparkConf().setAppName("My Word Count Job")
  val sc = new SparkContext(conf)
 val file = sc.textFile("/spark/fruits.txt")
  val words = file.flatMap(x => x.split(","))
  val wordsPair = words.map(x => (x, 1))
  val wordsCount = wordsPair.reduceByKey(_ + _)
  val sorted = wordsCount.sortBy(x => x._2, false)
    sorted.saveAsTextFile("/spark/eclipse_out")
}
Getting an error while execution ->
 
[cloudera@quickstart ~]$ spark-submit --master local --class WordCount wcspark.jar
Exception in thread "main" java.lang.NoSuchMethodException: WordCount.main([Ljava.lang.String;)
    at java.lang.Class.getMethod(Class.java:1665)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:716)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
 
Kindly help me out
 
2 REPLIES 2

Re: Error CLOUDERA - wordcounts

New Contributor

i believe that the error msg is very clear "java.lang.NoSuchMethodException: WordCount.main".

maybe you should find some other version of spark wordcount. "object WordCount extends App {" is the key which caused your trouble.

Re: Error CLOUDERA - wordcounts

Contributor

I tried to use main instead of extend App but still getting the same error.

Don't have an account?
Coming from Hortonworks? Activate your account here