First, thanks for following up. I sincerely appreciate the response. I figured out what's happening. To provide some context, I was trying to invoke some PySpark code from a (Scala) Python application, and getting the error message about the insecure gateway. It turns out this is relatively easy to fix, I just couldn't find any documentation talking about it anywhere. The driver application (which in my case is the Scala application) needs to generate or provide some type of shared secret, which Py4J refers to as the auth_token. Then it needs to provide that to the Python code, so that when the PySpark application needs to communicate through the Py4J gateway, it can use the same auth_token as a means of authentication. There is a very short code example of this on the Py4J website. It shows two snippets, one for the Java/Scala side (when creating the gateway server), and one for the Python side (when connecting to the gateway). Notice that both snippets refer to the same auth_token, and hence the authentication will succeed over a "secure" gateway. So essentially, what you need to do is: Generate or obtain some type of shared secret string to use as the auth_token In the Scala/Java side, pass this auth_token as a parameter when creating the GatewayServer When invoking the Python code, pass the auth_token to it in some secure manner In the Python code, use the auth_token that was passed as a parameter in the GatewayParameters This should set up the Py4J gateway in a secure manner and avoid the error (and also the need for PYSPARK_ALLOW_INSECURE_GATEWAY).
... View more
