PySpark source code bundled with CDH 6.3 Spark doesn't match Apache source

Question

I'm looking at the Spark parcel that is included with CDH 6.3, and seeing something strange. Specifically, the source code under python/pyspark doesn't seem to match up with what you see in the upstream (Apache) repository.

Take one file as an example.

/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/context.py

Jumping to line 114 of this file, we see:

        self._callsite = first_spark_call() or CallSite(None, None, None)
        if gateway is not None and gateway.gateway_parameters.auth_token is None:
            raise ValueError(
                "You are trying to pass an insecure Py4j gateway to Spark. This"
                " is not allowed as it is a security risk.")

Now, this file should come from Spark 2.4.0 (see both /opt/cloudera/parcels/CDH/lib/spark/RELEASE and /opt/cloudera/parcels/CDH/lib/spark/python/pyspark/version.py files, which indicate this). However, if you look at the upstream source, at the same line number, these lines don't exist. In fact, the source looks quite different. It has no such error referring to the insecure Py4j gateway.

Can anyone explain this discrepancy?

venkatsambath · Accepted Answer

@JeffEvans You are right. In CDH we cherry pick jiras to be included in our spark. Not all features available in upstream are expected to be present on CDH spark. The line number you quoted was added in this jira https://issues.apache.org/jira/browse/SPARK-1087 and is not back-ported to our spark code base. This is one of the reason we quote the following in our documentation

Although this document makes some references to the external Spark site, not all the features, components, recommendations, and so on are applicable to Spark when used on CDH. Always cross-check the Cloudera documentation before building a reliance on some aspect of Spark that might not be supported or recommended by Cloudera.

Hope this clarifies.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.