Support Questions
Find answers, ask questions, and share your expertise

Spark wordcount execution error

Spark wordcount execution error

Contributor

Hi

I am new in Spark. I am working on Standalone. I have downloaded Eclipse JDK 8 on my local and written a wordcount program in Scala :-

import org.apache.spark.SparkConf

import org.apache.spark.SparkContext

object WordCount extends App

{

val conf = new SparkConf().setAppName("My Word Count Job")

val sc = new SparkContext(conf)

val file = sc.textFile("/spark/fruits.txt")

val words = file.flatMap(x => x.split(","))

val wordsPair = words.map(x => (x, 1))

val wordsCount = wordsPair.reduceByKey(_ + _)

val sorted = wordsCount.sortBy(x => x._2, false)

sorted.saveAsTextFile("/spark/eclipse_out") }

Getting an error while execution -> spark-submit --master local --class WordCount wcspark.jar

Exception in thread "main" java.lang.NoSuchMethodException: WordCount.main([Ljava.lang.String;)
at java.lang.Class.getMethod(Class.java:1665)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:716)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Kindly help me out
1 REPLY 1

Re: Spark wordcount execution error

Explorer

You might wanna add a main method in your java class

object WordCount {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local").setAppName("WordCount")
    val sc = new SparkContext(conf)
    val data = sc.textFile("C:\\Users\\scala.txt")
    val result = data.flatMap(_.split(" ")).map(word => (word , 1)).reduceByKey(_ + _)
    result.collect.foreach(println)
  }