question Re: saveAsOrcFile is not a member of org.apache.spark.sql.DataFrame in Archives of Support Questions (Read Only)

saveAsOrcFile is not a member of org.apache.spark.sql.DataFrame

wei_yang80 — Sat, 13 Feb 2016 01:28:44 GMT

When I try Tutorial "A Lap around Apache Spark 1.3.1 with HDP 2.3" from sandbox, I encountered the problem:

scala> peopleSchemaRDD.saveAsOrcFile("people.orc")

<console>:41: error: value saveAsOrcFile is not a member of org.apache.spark.sql.DataFrame peopleSchemaRDD.saveAsOrcFile("people.orc") ^

Re: saveAsOrcFile is not a member of org.apache.spark.sql.DataFrame

bleonhardi — Sat, 13 Feb 2016 01:39:42 GMT

Are you sure that you are using the new sandbox and the Spark version is actually 1.3.1 or higher? It sounds like an error you would get in Spark 1.2

Re: saveAsOrcFile is not a member of org.apache.spark.sql.DataFrame

jstraub — Sat, 13 Feb 2016 01:43:31 GMT

. @wei yang Are you using Spark 1.3.1 or just the content of the tutorial? ORC support was added in Spark 1.4 (http://hortonworks.com/blog/bringing-orc-support-into-apache-spark/)

Try using the following command

myDataFrame.write.format("orc").save("some_name")

Re: saveAsOrcFile is not a member of org.apache.spark.sql.DataFrame

aervits — Sat, 13 Feb 2016 04:14:55 GMT

sc.parallelize(records).toDF().write.format("orc").save("people")

that method was refactored. There's a new way of writing ORC files. Convert your RDD to DataFrame with toDF() and then write it out as above.Try to use later versions of Spark.

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_spark-guide/content/ch_orc-spark.html

Re: saveAsOrcFile is not a member of org.apache.spark.sql.DataFrame

wei_yang80 — Sun, 14 Feb 2016 04:01:12 GMT

I'm using Spark 1.4.1, and the command "peopleSchemaRDD.write.format("orc").save("people.orc")" works !!!

Thank you very much !