#I have saved one RDD with 4 part files initially underneath of one directory.
But I have a use case That I need do separate each part file of a particular data set should save in different directories.
Similar solution is described here:
as a simpler alternative they also suggest
df.write.partitionBy("year", "month", "day").parquet("/path/to/output")
which will create directory structure for the partitioned columns of the dataframe.
*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.
Thank you for your information. But I need like below
I already have a directory contains four part files
merchant_table (main dir)
Again, I need to save the above one as separate four directories and one part file for that like below
merchant_table1 ( dir1)
merchant_table1 ( dir2)
merchant_table1 ( dir3)
merchant_table1 ( dir4)