Hi, @quangbilly79  Yes, you can continue to use HDFS normally while the Balancer is running. The Balancer only moves replicated block copies between DataNodes to even out disk usage; it does not modify the actual data files. Reads and writes are fully supported in parallel with balancing, and HDFS ensures data integrity through replication and checksums. The process may add some extra network and disk load, so you might see reduced performance during heavy balancing. There is no risk of data corruption caused by the Balancer. You don’t need to wait — it’s safe to continue your normal operations. ... View more

The node must have a NodeManager role to take part of the processing, Spark gateway, and Yarn Gateway ... View more

I've successfully setup Spark 3.3.0 on CDH 6.2 (we used YARN). Here are some important step 1. Back up the current spark come from Cloudera package (v2.4.0 I think) at /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark 2. Download the spark version from Spark homepage, for ex "spark-3.3.0-bin-hadoop3.tgz". Extract, delete old spark folder and replace with new spark folder (rename it to "spark") at /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark 3. Copy all the config files from old spark conf folder to the new spark conf folder   4. Copy the Yarn-related config file into spark conf folder too 4.1. Copy file spark-3.3.0-yarn-shuffle.jar from spark/yarn to spark/jars folder 5. Make some modifications to spark-default.conf file, mostly disable log and point to the right jar folder 6. Modify some yarn config like below (yarn-site.xml)     7. Restart the cluster and run spark-shell command. Run some queries for testing. You could modify the yarn-site.xml file in the spark conf folder directly to make sure.   ... View more