Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I am trying to create a table in spark. Below is the command. When I try to give format as CSV I am getting the below error

I am trying to create a table in spark. Below is the command. When I try to give format as CSV I am getting the below error

New Contributor

86550-error.png

rddstudent.write.format("csv").saveAsTable("Student_Spark2")

 "WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider csv. Persisting data source table `default`.`student_spark2` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive." Please let me know why I am not able to create a table using CSV as format
2 REPLIES 2
Highlighted

Re: I am trying to create a table in spark. Below is the command. When I try to give format as CSV I am getting the below error

@Sudharsan Ganeshkumar

AFAIK the csv format is not compatible between spark sql and hive serde and hence the error you are getting. A solution to this problem would be to:

1. create an external table pointing to the path where you will save the csv file

2. save the csv file instead of using saveAsTable function

spark.sql("CREATE EXTERNAL TABLE Student_Spark2(col1 int,col2 string) STORED AS TEXTFILE LOCATION '/path/in/hdfs'")
//later save
rddstudent.write.format("csv").save("/path/in/hdfs/student_spark2")

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: I am trying to create a table in spark. Below is the command. When I try to give format as CSV I am getting the below error

@Sudharsan Ganeshkumar If the above has helped, please take a moment to login and click the "accept" link on the answer.

Don't have an account?
Coming from Hortonworks? Activate your account here