question Pyspark Dataframe into Impala table ==> syntax error in Support Questions

Pyspark Dataframe into Impala table ==> syntax error

Ploeplse — Tue, 13 Sep 2022 09:25:25 GMT

Dear all,

I am trying to load a PySpark Dataframe into an Impala table, using the jdbc connector. However, the df.write statement fails, because the "Create table" - statement that is generated contains quotation marks for the column names:

Do you have any idea how to get rid of these quotation marks? If not, what would be a different approach to load a dataframe into an Impala table?

I also tried

spark.sql('select identifier_id as identifier from tempView').write.jdbc(...), but here I am getting the error "File /tmp/hive does not exist".

Thanks a lot in advance for any help!

Re: Pyspark Dataframe into Impala table ==> syntax error

RangaReddy — Wed, 14 Sep 2022 07:54:10 GMT

Hi @Ploeplse

Could you please share reproducible sample code and impala tab creation script?

Re: Pyspark Dataframe into Impala table ==> syntax error

VidyaSargur — Thu, 22 Sep 2022 07:51:23 GMT

HI @Ploeplse, If you are still experiencing the issue, can you provide the information @RangaReddy has requested?

Re: Pyspark Dataframe into Impala table ==> syntax error

RangaReddy — Tue, 11 Oct 2022 09:13:35 GMT

Hi @Ploeplse

Still, if you are facing the issue, could you share the requested information (i.e code and impala table creation script)