I am trying to create a table in spark. Below is the command. When I try to give format as CSV I am getting the below error

[#1]

rddstudent.write.format("csv").saveAsTable("Student_Spark2")

 "WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider csv. Persisting data source table `default`.`student_spark2` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive." Please let me know why I am not able to create a table using CSV as format

[#2]

@Sudharsan Ganeshkumar

AFAIK the csv format is not compatible between spark sql and hive serde and hence the error you are getting. A solution to this problem would be to:

1. create an external table pointing to the path where you will save the csv file

2. save the csv file instead of using saveAsTable function

spark.sql("CREATE EXTERNAL TABLE Student_Spark2(col1 int,col2 string) STORED AS TEXTFILE LOCATION '/path/in/hdfs'")
//later save
rddstudent.write.format("csv").save("/path/in/hdfs/student_spark2")

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

[#3]

@Sudharsan Ganeshkumar If the above has helped, please take a moment to login and click the "accept" link on the answer.