question Re: Reading 1000 tables from RDBMS via pyspark and create parquet tables in hive in Support Questions

Reading 1000 tables from RDBMS via pyspark and create parquet tables in hive

Meher — Fri, 16 Sep 2022 14:02:21 GMT

Team,

I have 1000 tables in my source RDBMS and I would like to get them migrated to hive using pyspark

I read through documentation and found that below two commands would help. Is there a way I can loop these two commands 1000 times if I have all the list of tables in a python array?

arr = ("table1","table2")

for x in arr:

df = spark.read.format("jdbc").blah.blah

data.write.saveAsTable.blah.blah

If someone has a working solution for this could you please share. I tried but it is not throwing any error but at same time not writing anything.

Thanks

Meher

Re: Reading 1000 tables from RDBMS via pyspark and create parquet tables in hive

Meher — Tue, 08 Jan 2019 17:11:30 GMT

I'm able to get this working. Will close this post.

Thanks,

Meher

Re: Reading 1000 tables from RDBMS via pyspark and create parquet tables in hive

cjervis — Wed, 09 Jan 2019 13:57:16 GMT

@Meher I am happy to see that you resolved your issue. Would you mind sharing how you solved it in case someone else encounters the same situation?