I have a hive on spark job which is taking too many res sources at the peak 1tb it has a few joins which causes the performance to lag. (It is only joining on a few million records not more than that)
How can I improve the performance with various configs in hive to
1. Reduce the resources consumed by this job is their away to limit the vcores and executor memory so that is doesnt take all the cluster ressources?
2. Are there any hive on spark specific configs to modify for this case lots of joins with say 5m records.