question pyspark crashes when running locally but works on a cluster in Support Questions

question pyspark crashes when running locally but works on a cluster in Support Questions https://community.cloudera.com/t5/Support-Questions/pyspark-crashes-when-running-locally-but-works-on-a-cluster/m-p/51720#M33386 If one run pyspark without arguments on the gateway node on which CM is installed, one gets:=========================$ pyspark Python 2.7.5 (default, Nov 20 2015, 02:00:19)   [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Error: Cluster deploy mode is not applicable to Spark shells. Run with --help for usage help or --verbose for debug output Traceback (most recent call last):  File "/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/spark/python/pyspark/shell.py", line 43, in <module>    sc = SparkContext(pyFiles=add_files)  File "/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/spark/python/pyspark/context.py", line 112, in __init__    SparkContext._ensure_initialized(self, gateway=gateway)  File "/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/spark/python/pyspark/context.py", line 245, in _ensure_initialized    SparkContext._gateway = gateway or launch_gateway()  File "/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/spark/python/pyspark/java_gateway.py", line 94, in launch_gateway    raise Exception("Java gateway process exited before sending the driver its port number") Exception: Java gateway process exited before sending the driver its port number =========================  However, it works on the cluster: =========================[ivy2@md01 ~]$ pyspark --master=yarn Python 2.7.5 (default, Nov 20 2015, 02:00:19)   [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). 17/03/02 14:54:04 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. Welcome to      ____              __     / __/__  ___ _____/ /__    _\ \/ _ \/ _ `/ __/  '_/   /__ / .__/\_,_/_/ /_/\_\   version 1.6.0      /_/ Using Python version 2.7.5 (default, Nov 20 2015 02:00:19) SparkContext available as sc, HiveContext available as sqlContext. >>> ========================= Any ideas why? Is this warning OK: "WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. "?    Fri, 16 Sep 2022 11:11:04 GMT IgorYakushin 2022-09-16T11:11:04Z