Support Questions
Find answers, ask questions, and share your expertise

Setup HDP 2.4 from scratch with SparkR, requirements

Setup HDP 2.4 from scratch with SparkR, requirements

Explorer

Hi, If I install HDP 2.4.x from scratch, and I want to use SparkR, do I need to install also an R distribution on the workers node?

Without R I'm able to use spark-shell from a node outside the cluster and everithing seems good (run Pi Estimation example without problem).

I am also able to use sparklyr library and do stuff, but if I understood well, it's because the sparklyr library perform some kind of translation of R code in something like scala.

When I tried to use SparkR library, it seems like it search the Rscript executable on the cluster. Do I need to install it on the workers or master node of the cluster? Or it's a problem with the cluster configuration? Or is a problem with the "submitter" outside of the cluster?

Bests,

Sergio L.

2 REPLIES 2

Re: Setup HDP 2.4 from scratch with SparkR, requirements

Expert Contributor

Re: Setup HDP 2.4 from scratch with SparkR, requirements

Expert Contributor