Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How Do You Install Python Packages in HDP Spark Cluster?

Highlighted

How Do You Install Python Packages in HDP Spark Cluster?

New Contributor

Hi,

I can ssh into my HDP cluster servers as user 'cloudbreak', but I cannot sudo or su to root in order to install python modules and packages I need for pyspark. I spent time searching for answers but came up with nothing. Any help appreciated.

3 REPLIES 3
Highlighted

Re: How Do You Install Python Packages in HDP Spark Cluster?

Mentor

Cloudbreak uses docker containers, just ssh to machine is not enough.

Highlighted

Re: How Do You Install Python Packages in HDP Spark Cluster?

New Contributor

Ahh yes. But question still remains how would one install python packages in the spark worker dockers such that they survive restart etc? I couldn't find any documentation on it.

Highlighted

Re: How Do You Install Python Packages in HDP Spark Cluster?

Mentor

@Darren Govoni I haven't touched Cloudbreak in a while, 1.2 version was just published and it introduced a concept of recipes. You can add your own custom scripts/jars etc to cluster. I believe you can add to an existing cluster as well. Try it out and let us know. http://sequenceiq.com/cloudbreak-docs/latest/recipes/

Don't have an account?
Coming from Hortonworks? Activate your account here