question Re: How to reduce the small problem in spark using coalesce or otherwise? in Archives of Support Questions (Read Only)

How to reduce the small problem in spark using coalesce or otherwise?

Former Member — Tue, 18 Jul 2017 04:11:30 GMT

When I insert my dataframe into a table it creates some small files.

One solution I had was to use to coalesce to one file but this greatly slows down the code. I am looking at a way to either improve this by somehow speeding it up while still coalescing to 1.

Like this

df_expl.coalesce(1)
  .write.mode("append")
  .partitionBy("p_id")
  .parquet(expl_hdfs_loc)

Or I am open to another solution.

Re: How to reduce the small problem in spark using coalesce or otherwise?

mqureshi — Tue, 18 Jul 2017 08:46:40 GMT

Also use DISTRIBUTE BY so data for same partition goes to same reducer.

Re: How to reduce the small problem in spark using coalesce or otherwise?

Former Member — Wed, 19 Jul 2017 00:08:28 GMT

Thanks for the response . I am sorry I dont really understand what you mean could you please provide an example?

Re: How to reduce the small problem in spark using coalesce or otherwise?

mqureshi — Wed, 19 Jul 2017 00:39:32 GMT

Please see the following link. In your code, you'll need to do a "repartition". What I am trying to say is if you force more data to same reducer, you will create less files. Call repartition function on some key where data for that key will land in same partition.

https://dzone.com/articles/optimize-spark-with-distribute-by-cluster-by

Re: How to reduce the small problem in spark using coalesce or otherwise?

Former Member — Wed, 09 Aug 2017 20:49:16 GMT

Thanks for the response and the link is very useful. I had one more question if I do df.repartition(1).write will this only run on one node or will this run in a distributed way but only create one file. Which is the problem I face with coalesce when I right to parquet it writes one file but only on one node and I lose the whole distributed advantages.

Is there any way I could do something like this

joinedDF.repartition(col("partitionCol)).coalesce(1).write.mode("append").partitionBy("partitionCol").parquet(esdLocation) but it should only coalesce per partition.

Thanks

Re: How to reduce the small problem in spark using coalesce or otherwise?

brijesh_jaggi1 — Fri, 25 May 2018 08:04:37 GMT

https://community.hortonworks.com/questions/8010/hives-alter-table-partition-concatenate-not-workin.html

this solution seems better...