How Hadoop works with agile and scrum methodologies? How does development, QA, Integration and Production release workflow involved in Hadoop projects ?

[#1]

Hi,

I'm trying to implement Hadoop project and i'm researching on how the SDLC workflow involved in the Hadoop project.

Thanks,

[#2 · Accepted solution]

I don't believe hardware or infrastructure setup should follow any such workflow.

Maybe you need separate environments to isolate workloads, but other than that, it's the code itself that follows development patterns as anything else.

Examples of such code could include MapReduce, Hive scripts, Oozie jobs, Spark processes, NiFi dataflows, etc.

In terms of MapReduce or Spark, you can use CI/CD processes to build code and push it to HDFS, and submit it to YARN to run once, or submit it to Oozie to run on a schedule.

Hadoop itself just offers HDFS, YARN, and MapReduce. It's everything else that is very specific to your needs and processes.

[#3]

@Jordan Moore

Thanks for the update !!!

We are also working in the same fashion as you said, but I thought that other companies might be following agile/scrum methodologies for the Hadoop Development.

Also, I have one more question; How is stand-up meetings or client interaction process done in big Data projects ?

[#4]

@Rakesh AN

I have worked for at least three companies trying to follow Agile/Scrum, and their development cycles of code does follow it.

It's hard to upgrade hundreds of Hadoop nodes and software versions, make sure they all work with other components of the cluster, all without breaking other pieces in two-week sprints, though.

Stand up meetings are all about perception management between team members and management. It again, has no special relationship or difference whether it is Hadoop development, web or mobile development, etc.