Continuous Integration/Delivery of HDP clusters on Cloudbreak

[#1]

Hi,

I are running a number of HDP clusters on Cloudbreak. The code is residing on GitHub (Spark Python and Scala), we would like to setup a Continuous integration and Delivery pipeline from Github.

I have searched quite a bit for recommended practices for this but to no avail. At this point, I am thinking about using Cloudbreak Command Line (CLI) to create the scripts needed and run them from github.

1. Any better recommendations?

2. Does that seem like a valid approach?

3. How would I deal with security since I would need to store Cloudbreak credentials on Github to run my scripts?

Any input is appreciated.

Thx

[#2 · Accepted solution]

Hi,

@automating cluster creation
Using the CLI for automated cluster setup is a valid approach.

@security
Storing secure data in git repository is not a good idea.

I would assume that you have Cloudbreak deployed and a CI job would have a configured cli to use it. I would also assume that this machine is secured as if you have access to this machine, you basically have access to everything cloudbreak stores.

The configuration for CLI is located usually in ~/cb/config, and has the following

testenv1:
  username: user1
  password: user1pw
  server: 10.0.10.12

I'm not sure what kind of CI framework you plan to use (jenkins, teamcity, spinnaker), but basically you could have a CLI installed on an agent or a docker container having the cli and an example configuration file injected from the context of the CI build agent.
Another idea is to have the encrypted cli config stored in github, and have the agent the decryption key, and apply the correct config.