Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark wordcount execution error

Highlighted

Spark wordcount execution error

Contributor

Hi

I am new in Spark. I am working on Standalone. I have downloaded Eclipse JDK 8 on my local and written a wordcount program in Scala :-

import org.apache.spark.SparkConf

import org.apache.spark.SparkContext

object WordCount extends App

{

val conf = new SparkConf().setAppName("My Word Count Job")

val sc = new SparkContext(conf)

val file = sc.textFile("/spark/fruits.txt")

val words = file.flatMap(x => x.split(","))

val wordsPair = words.map(x => (x, 1))

val wordsCount = wordsPair.reduceByKey(_ + _)

val sorted = wordsCount.sortBy(x => x._2, false)

sorted.saveAsTextFile("/spark/eclipse_out") }

Getting an error while execution -> spark-submit --master local --class WordCount wcspark.jar

Exception in thread "main" java.lang.NoSuchMethodException: WordCount.main([Ljava.lang.String;)
at java.lang.Class.getMethod(Class.java:1665)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:716)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Kindly help me out
1 REPLY 1

Re: Spark wordcount execution error

New Contributor

You might wanna add a main method in your java class

object WordCount {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setMaster("local").setAppName("WordCount")
    val sc = new SparkContext(conf)
    val data = sc.textFile("C:\\Users\\scala.txt")
    val result = data.flatMap(_.split(" ")).map(word => (word , 1)).reduceByKey(_ + _)
    result.collect.foreach(println)
  }
Don't have an account?
Coming from Hortonworks? Activate your account here