In a pyspark code which reads a hdfs path in it's respective format(text, orc, parquet) and writes it in parquet format in ozone path. Data is huge. 1) How to do resource calculations for the pyspark job Like no. of cores, no. of executors, memory allocation   2) Is there a way we can dynamically read the data from hdfs by adjusting according to it's file type.   3) What should be the optimal solution and approach for the data movement and what input mappings the approach should use. ... View more