question Re: Spark-sklearn integration in Archives of Support Questions (Read Only)

Spark-sklearn integration

Chandra — Thu, 03 Aug 2017 04:11:15 GMT

Hi,

We have a Hadoop on-premise cluster and are planning to integrate spark with scikit learn using the spark-sklearn package. Can you please let me know if we need to install sklearn and spark-sklearn package in all nodes or just the node where spark2-history server has been installed. We will be using yarn for resource allocation.

Thanks,

Chandra

Re: Spark-sklearn integration

dzaratsian — Thu, 03 Aug 2017 06:22:32 GMT

@chandramouli muthukumaran

You'll want to install sklearn (pip install -U scikit-learn) and spark-sklearn on all datanodes of the cluster, as well as other relevant python packages such as numpy, scipy, etc. I'd also recommend using YARN as the resource manager, so you are on the right path there. Hope this helps!

Re: Spark-sklearn integration

Chandra — Fri, 04 Aug 2017 03:22:22 GMT

Thanks much for your response.