Reply
Explorer
Posts: 6
Registered: ‎01-09-2017

PySpark 2.1 Error in util.py

Hey there,

 

we found a little Error in /opt/cloudera/parcels/SPARK2/lib/spark2/python/pyspark/ml/util.py there is a "import warnings" line missing 

 

This issue was fixed a few month ago in Spark:

 

https://github.com/apache/spark/pull/16846

 

Best,

Florian

Explorer
Posts: 6
Registered: ‎01-09-2017

Re: PySpark 2.1 Error in util.py

also...

line 223 of the Same file could need some hash.

 

#Remove the last package name "pipeline" for Pipeline and PipelineModel.

Cloudera Employee
Posts: 31
Registered: ‎11-16-2015

Re: PySpark 2.1 Error in util.py

The fix for SPARK-19506 is incorporated in Cloudera Distribution of Apache Spark 2.2 release 1

For a full list of fixed issues, see here

For downloading Cloudera Distribution of Apache Spark 2.2 release 1, see here

 

Explorer
Posts: 6
Registered: ‎01-09-2017

Re: PySpark 2.1 Error in util.py

So this actualy means, that we have to choose either a broken PySpark 2.1 or to Upgrade the JDK to 1.8, yeah

Highlighted
Cloudera Employee
Posts: 31
Registered: ‎11-16-2015

Re: PySpark 2.1 Error in util.py

Yes, JDK 8 is a requirement for Spark 2.2 (which has the fix for SPARK-19506).

Announcements