Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

PySpark 2.1 Error in util.py

PySpark 2.1 Error in util.py

Explorer

Hey there,

 

we found a little Error in /opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/ml/util.py there is a "import warnings" line missing 

 

This issue was fixed a few month ago in Spark:

 

https://github.com/apache/spark/pull/16846

 

Best,

Florian

4 REPLIES 4

Re: PySpark 2.1 Error in util.py

Explorer

also...

line 223 of the Same file could need some hash.

 

#Remove the last package name "pipeline" for Pipeline and PipelineModel.

Re: PySpark 2.1 Error in util.py

Expert Contributor

The fix for SPARK-19506 is incorporated in Cloudera Distribution of Apache Spark 2.2 release 1

For a full list of fixed issues, see here

For downloading Cloudera Distribution of Apache Spark 2.2 release 1, see here

 

Re: PySpark 2.1 Error in util.py

Explorer

So this actualy means, that we have to choose either a broken PySpark 2.1 or to Upgrade the JDK to 1.8, yeah

Highlighted

Re: PySpark 2.1 Error in util.py

Expert Contributor

Yes, JDK 8 is a requirement for Spark 2.2 (which has the fix for SPARK-19506).