Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Tutorial Exercise 3 - not found: value sc

Tutorial Exercise 3 - not found: value sc

New Contributor

The Scala code in this exercise does not compile:

 

https://www.cloudera.com/developers/get-started-with-hadoop-tutorial/exercise-3.html

 

I get the error:

 

<console>:30: error: not found: value sc
          return sc.newAPIHadoopRDD(job.getConfiguration,

 

Where is the variable 'sc' initialized? There are many errors in these tutorials which is a bit frustrating, I hope someone can tell me how to overcome it, thanks.

 

I am running the latest CDH Quickstart docker image.

1 REPLY 1

Re: Tutorial Exercise 3 - not found: value sc

Champion

@rupertlssmith

 

 

You have to initialize sc depends upon how you are executing your code. If you are using spark-shell command line then you don't need to initilize sc as it will be initialized by default when you login but if you are developing code in other 3rd party tools and executing then you have to initilialize as follows:

 

You can add the below lines before you call rddFromParquetHdfsFile

 

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

val conf = new SparkConf().setAppName("your topic").setMaster("yarn-client")
val sc = new SparkContext(conf)