Reply
Explorer
Posts: 22
Registered: ‎12-10-2014

How do you do Continuous Delivery with the Cloudera stack?

Hi,

I'm not entirely sure if this question should be here or in the getting started board.  If I'm in the wrong place please let me know.

 

Anyway, my organisation has Cloudera as an "innovation" deployment where everyone develops against a single instance of most of the components of the Cloudera stack.  There's a lot of manual activity to get software written, tested and artefacts prepared for potential deployment.

 

Naturally, there's a desire to shift to a continuous delivery method as we grow into a "production" deployment.

 

How do you do continuous delivery?

 

At the moment I'm thinking of an environment where a developer has a personal development environment (i.e. Cloudera instance and integrated development environment) hydrated (and dehydrated) and configured on demand with all artefacts held in a permenant Team Foundation Server source control repository.  Code is created and unit tests are run in this space: small scale and no integration.

 

On a half day schedule the integration environment will be hydrated, configured and artefacts generated.  The integration test suite kicks off and runs to produce reports for developers, which are checked back into source control.  Finally the integration envirenment is dehydrated.

 

If all the tests succeed the next level of assurance is performed in its own environment, etc; until production deployment (manual step).

 

Is this practical/doable?

 

I'm also interested in how you do testing and integrate source control; but I'll hold that conversation for another thread topic.

 

Thanks,

Ty

Explorer
Posts: 7
Registered: ‎05-08-2015

Re: How do you do Continuous Delivery with the Cloudera stack?

I have the same question and have been unable to find any information on this.  Hopefully this will kick start a conversation with how others are managing their code within a Hadoop environment.  

Cloudera Employee
Posts: 435
Registered: ‎07-12-2013

Re: How do you do Continuous Delivery with the Cloudera stack?

Depending on which parts of the stack you use, it might be useful to learn
how other Apache projects that build against Hadoop do QA. A lot of
projects use MiniCluster (
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CLIMiniCluster.html)
to do a layer of lightweight testing against YARN & HDFS. One of the
intended purposes of the QuickStart VM is to use as a fairly realistic test
environment / appliance for the whole stack before deploying code on a full
cluster.
Explorer
Posts: 15
Registered: ‎01-31-2019

Re: How do you do Continuous Delivery with the Cloudera stack?

...and that last link is also dead.

 

So much of this stuff is roll-your-own. The way forward will be to see how TFS is integrated into existing development platforms (i.e. how everyone is already doing it) and then try and replicate that.