Using Hortonworks Sandbox, I am setting up SparkR in both RStudio and Zeppelin. This below code works properly in RStudio and SparkR shell but not in Zeppelin, please have a look: if (nchar(Sys.getenv("SPARK_HOME")) < 1) {
Sys.setenv(SPARK_HOME = "/usr/hdp/2.5.0.0-1245/spark")
}
library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sc <- sparkR.init(master = "local[*]", sparkEnvir = list(spark.driver.memory="2g"),sparkPackages="com.databricks:spark-csv_2.10:1.4.0")
sqlContext <- sparkRSQL.init(sc)
train_df <- read.df(sqlContext,"/tmp/first_8.csv","csv", header = "true", inferSchema = "true") But when I do this in Zeppelin using livy.spark interpreter, I get ClassNotFound Exception: java.lang.ClassNotFoundException: Failed to find data source: csv. Please find packages at http://spark-packages.org I am also importing the dependencies using dep interpreter - %dep
z.reset()
z.load("com.databricks:spark-csv_2.10:1.4.0") But this seems to make no impact I guess. I have also tried manually copying spark-csv_2.10-1.4.0.jar to /usr/hdp/2.5.0.0-1245/spark/lib, but it is not working. Has anyone experienced this before? Thanks in advance
... View more
