Hello @thoufeeq1218, We understand that you have configured spark.blockManager.driver.port and spark.blockManager.port, but Spark may still attempt to use random ports due to the following reasons: - Why is Spark using random ports? Spark uses additional ports beyond BlockManager for communication between the driver and executors. The random port seen (59698) is from the ephemeral port range (1024–65535) and could be assigned due to: spark.driver.port (default: random) spark.executor.port (default: random) - How to restrict Spark to specific ports? Explicitly set spark.driver.port to ensure the driver listens on a fixed port: --conf spark.driver.port=21800 Ensure spark.blockManager.port is available If 21700 is occupied, Spark will fall back to a random port. - Understanding spark.port.maxRetries If spark.port.maxRetries > 0 (default: 16), Spark will try additional ports within the ephemeral range. If spark.port.maxRetries = 0, Spark will fail immediately if the specified port is unavailable. - Executors and Dynamic Ports Executors start/stop dynamically and may request random ports. If you must prevent Spark from using random ephemeral ports, use the following settings: --conf spark.driver.port=21800 --conf spark.blockManager.driver.port=21750 --conf spark.blockManager.port=21700 --conf spark.executor.port=21810 --conf spark.port.maxRetries=0 These settings can be applied at the job level or in the Spark configuration file. Note: If a port is already used by a running job, a new job may fail due to a port conflict. If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post. Thank you.
... View more
