Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 11-28-2016 11:36 PM
Using Hortonworks Sandbox, I am setting up SparkR in both RStudio and Zeppelin. This below code works properly in RStudio and SparkR shell but not in Zeppelin, please have a look:
if (nchar(Sys.getenv("SPARK_HOME")) < 1) {
Sys.setenv(SPARK_HOME = "/usr/hdp/2.5.0.0-1245/spark")
}
library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib")))
sc <- sparkR.init(master = "local[*]", sparkEnvir = list(spark.driver.memory="2g"),sparkPackages="com.databricks:spark-csv_2.10:1.4.0")
sqlContext <- sparkRSQL.init(sc)
train_df <- read.df(sqlContext,"/tmp/first_8.csv","csv", header = "true", inferSchema = "true")But when I do this in Zeppelin using livy.spark interpreter, I get ClassNotFound Exception:
java.lang.ClassNotFoundException: Failed to find data source: csv. Please find packages at http://spark-packages.org
I am also importing the dependencies using dep interpreter -
%dep
z.reset()
z.load("com.databricks:spark-csv_2.10:1.4.0")But this seems to make no impact I guess. I have also tried manually copying spark-csv_2.10-1.4.0.jar to /usr/hdp/2.5.0.0-1245/spark/lib, but it is not working. Has anyone experienced this before? Thanks in advance
Created 12-06-2016 10:34 PM
Got it working finally, thanks to @Robert Hryniewicz. Go to interpreter settings page and add the new property under livy settings - livy.spark.jars.packages and the value com.databricks:spark-csv_2.10:1.4.0. Restart the interpreter and retry the query.
Created 12-05-2016 04:11 AM
Please specify com.databricks:spark-csv_2.10:1.4.0 in the interpreter setting page
Created 12-05-2016 07:52 PM
@jzhang,should I add it in the livy interpreter?
Created 12-06-2016 08:17 PM
I tried that, but it didn't work
Created 12-06-2016 10:34 PM
Got it working finally, thanks to @Robert Hryniewicz. Go to interpreter settings page and add the new property under livy settings - livy.spark.jars.packages and the value com.databricks:spark-csv_2.10:1.4.0. Restart the interpreter and retry the query.