Created on 08-17-2018 08:35 AM - edited 08-17-2019 07:05 PM
rddstudent.write.format("csv").saveAsTable("Student_Spark2") "WARN HiveExternalCatalog: Couldn't find corresponding Hive SerDe for data source provider csv. Persisting data source table `default`.`student_spark2` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive." Please let me know why I am not able to create a table using CSV as format
AFAIK the csv format is not compatible between spark sql and hive serde and hence the error you are getting. A solution to this problem would be to:
1. create an external table pointing to the path where you will save the csv file
2. save the csv file instead of using saveAsTable function
spark.sql("CREATE EXTERNAL TABLE Student_Spark2(col1 int,col2 string) STORED AS TEXTFILE LOCATION '/path/in/hdfs'") //later save rddstudent.write.format("csv").save("/path/in/hdfs/student_spark2")
*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.