I'm having a Hive table emp1 with 100 partitions in Text format. I want Spark to read emp table based on partitions bases and write to EMP2 in parquet format. How to achieve 1) 10 Partition Read from EMP and parallel write to emp2. Making sure multiple executors are running and processing the data in parallel to convert data in short period. I don't want to use HDFS path. ... View more

In a pyspark code which reads a hdfs path in it's respective format(text, orc, parquet) and writes it in parquet format in ozone path. Data is huge. 1) How to do resource calculations for the pyspark job Like no. of cores, no. of executors, memory allocation   2) Is there a way we can dynamically read the data from hdfs by adjusting according to it's file type.   3) What should be the optimal solution and approach for the data movement and what input mappings the approach should use. ... View more