Created 08-22-2017 10:23 AM
Does anyone know what the procedure is for installing graphframes for pyspark2 on
SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931?
Or more generally, how to instal any package for spark in cloudera?.
Created 08-24-2017 10:16 AM
You can use "--packages" with the shell to include any additional 3rd party packages you'd want. For graphframes , please refer to https://spark-packages.org/package/graphframes/graphframes, pick the suitable spark version and install it.
Since you are on Spark ver 2.0 and assuming the scala ver 2.11, you should likely use 0.5.0-spark2.0-s_2.11
# pyspark2 --packages graphframes:graphframes:0.5.0-spark2.0-s_2.11