I am setting up resource pools for the first time and I want to make sure I am able to accomplish what I set out to do.
My cluster is mostly Spark/YARN & Impala. I would like both services to fully utilize memory when the other is idle. Currently, each service has a hard memory limit configuration. I would like that to limit to be fluid based on utilization of the opposing service. If Impala is idle, Spark should be allowed to grab gobs of memory.
I originally thought I could setup dynamic service pools and be done. As I got into various documentation, I am second guessing that. Some documentation leads me to believe that static pools need to be configured as the parent of the dynamic pools, but static pools do not allow resource sharing across services.
If I omit the staic pool and setup dynamic pools for YARN & Impala will I accomplish my goal?