Hello @Jack_sparrow, Glad to see you again in the forums. 1. The resource allocation is something kind of complicated to tell, because it depends in a lot of factors. Take in mind how big is your data, how much memory you have on the cluster, do not forget the overhead and other things. There is very useful information here: https://docs.cloudera.com/cdp-private-cloud-base/7.3.1/tuning-spark/topics/spark-admin-tuning-resource-allocation.html Under that parent section there is more tuning suggestions on each topic. 2. From the second option I understand that you want to read the data separately using each type. That should be possible with something like this: if input_path.endswith(".parquet"): df = spark.read.parquet(input_path) elif input_path.endswith(".orc"): df = spark.read.orc(input_path) elif input_path.endswith(".txt") or input_path.endswith(".csv"): df = spark.read.text(input_path) # o .csv con opciones else: raise Exception("Unsupported file format") Then, you can handle each data in a separate way. 3. The data movement should avoid going to the driver, to avoid issues and extra work, so collect() or .toPandas() are not the best options. If you want to move data without transformations, distcp should be a good option. To write you can use this: df.write.mode("overwrite").parquet("ofs://ozone/path/out") And other suggestions can be tuning the partitions with "spark.sql.files.maxPartitionBytes" and change the compression to snappy using "spark.sql.parquet.compression.codec".
... View more
