Skip to main content
Meher
Gold ⭐️
January 8, 2019
Solved

Reading 1000 tables from RDBMS via pyspark and create parquet tables in hive

Team,

 

I have 1000 tables in my source RDBMS and I would like to get them migrated to hive using pyspark

 

I read through documentation and found that below two commands would help. Is there a way I can loop these two commands 1000 times if I have all the list of tables in a python array?

 

arr = ("table1","table2")

for x in arr:

            df = spark.read.format("jdbc").blah.blah 

 

            data.write.saveAsTable.blah.blah

 

If someone has a working solution for this could you please share. I tried but it is not throwing any error but at same time not writing anything.

 

Thanks

Meher

 

Meher
MeherAnswer
Gold ⭐️
January 8, 2019

I'm able to get this working. Will close this post. 

 

Thanks,

Meher

2 replies

Meher
MeherAuthorAnswer
Gold ⭐️
January 8, 2019

I'm able to get this working. Will close this post. 

 

Thanks,

Meher

Translate
cjervis
Community Manager
Community Manager
January 9, 2019

@Meher I am happy to see that you resolved your issue. Would you mind sharing how you solved it in case someone else encounters the same situation?

Translate
Keep the questions coming