Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

pyspark <dataframe>.write.saveAsTable not creating table in hive

Highlighted

pyspark <dataframe>.write.saveAsTable not creating table in hive

New Contributor

Hi Experts,

I am trying to save a dataframe as a hive table using <dataframe>.write.saveAsTable method using pyspark.

The command gives warning, creates directory in dfs but not the table in hive metastore.

I was reading many old posts which say that this command doesnt work and I need to create the table manually pointing to the directory created by the above pyspark command.

Just wanted to ask in this forum if this is true and this limitation exist with current version of cloudera.

 

 

 

 

1 REPLY 1

Re: pyspark <dataframe>.write.saveAsTable not creating table in hive

Champion

It is late so I am not recalling the specifics but yes, I recommend always creating the Hive table definition outside of Spark.  I vaguelyl recall that if you let Spark create it other services can't use it and Spark having issues translating the metadata correctly to the Hive specification.