question Not able to create SparkR dataframe using read.df in Archives of Support Questions (Read Only)

Not able to create SparkR dataframe using read.df

mrizvi — Tue, 29 Nov 2016 07:36:55 GMT

Using Hortonworks Sandbox, I am setting up SparkR in both RStudio and Zeppelin. This below code works properly in RStudio and SparkR shell but not in Zeppelin, please have a look:

if (nchar(Sys.getenv("SPARK_HOME")) < 1) {
  Sys.setenv(SPARK_HOME = "/usr/hdp/2.5.0.0-1245/spark")
}
library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sc <- sparkR.init(master = "local[*]", sparkEnvir = list(spark.driver.memory="2g"),sparkPackages="com.databricks:spark-csv_2.10:1.4.0")
sqlContext <- sparkRSQL.init(sc)
train_df <- read.df(sqlContext,"/tmp/first_8.csv","csv", header = "true", inferSchema = "true")

But when I do this in Zeppelin using livy.spark interpreter, I get ClassNotFound Exception:

java.lang.ClassNotFoundException: Failed to find data source: csv. Please find packages at http://spark-packages.org

I am also importing the dependencies using dep interpreter -

%dep
z.reset()
z.load("com.databricks:spark-csv_2.10:1.4.0")

But this seems to make no impact I guess. I have also tried manually copying spark-csv_2.10-1.4.0.jar to /usr/hdp/2.5.0.0-1245/spark/lib, but it is not working. Has anyone experienced this before? Thanks in advance

Re: Not able to create SparkR dataframe using read.df

jzhang — Mon, 05 Dec 2016 12:11:41 GMT

Please specify com.databricks:spark-csv_2.10:1.4.0 in the interpreter setting page

Re: Not able to create SparkR dataframe using read.df

mrizvi — Tue, 06 Dec 2016 03:52:12 GMT

@jzhang,should I add it in the livy interpreter?

Re: Not able to create SparkR dataframe using read.df

mrizvi — Wed, 07 Dec 2016 04:17:29 GMT

I tried that, but it didn't work

Re: Not able to create SparkR dataframe using read.df

mrizvi — Wed, 07 Dec 2016 06:34:29 GMT

Got it working finally, thanks to @Robert Hryniewicz. Go to interpreter settings page and add the new property under livy settings - livy.spark.jars.packages and the value com.databricks:spark-csv_2.10:1.4.0. Restart the interpreter and retry the query.