Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How Hadoop works with agile and scrum methodologies? How does development, QA, Integration and Production release workflow involved in Hadoop projects ?

Solved Go to solution
Highlighted

How Hadoop works with agile and scrum methodologies? How does development, QA, Integration and Production release workflow involved in Hadoop projects ?

New Contributor

Hi,

I'm trying to implement Hadoop project and i'm researching on how the SDLC workflow involved in the Hadoop project.

Thanks,

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How Hadoop works with agile and scrum methodologies? How does development, QA, Integration and Production release workflow involved in Hadoop projects ?

Super Collaborator

I don't believe hardware or infrastructure setup should follow any such workflow.

Maybe you need separate environments to isolate workloads, but other than that, it's the code itself that follows development patterns as anything else.

Examples of such code could include MapReduce, Hive scripts, Oozie jobs, Spark processes, NiFi dataflows, etc.

In terms of MapReduce or Spark, you can use CI/CD processes to build code and push it to HDFS, and submit it to YARN to run once, or submit it to Oozie to run on a schedule.

Hadoop itself just offers HDFS, YARN, and MapReduce. It's everything else that is very specific to your needs and processes.

3 REPLIES 3

Re: How Hadoop works with agile and scrum methodologies? How does development, QA, Integration and Production release workflow involved in Hadoop projects ?

Super Collaborator

I don't believe hardware or infrastructure setup should follow any such workflow.

Maybe you need separate environments to isolate workloads, but other than that, it's the code itself that follows development patterns as anything else.

Examples of such code could include MapReduce, Hive scripts, Oozie jobs, Spark processes, NiFi dataflows, etc.

In terms of MapReduce or Spark, you can use CI/CD processes to build code and push it to HDFS, and submit it to YARN to run once, or submit it to Oozie to run on a schedule.

Hadoop itself just offers HDFS, YARN, and MapReduce. It's everything else that is very specific to your needs and processes.

Re: How Hadoop works with agile and scrum methodologies? How does development, QA, Integration and Production release workflow involved in Hadoop projects ?

New Contributor

@Jordan Moore

Thanks for the update !!!

We are also working in the same fashion as you said, but I thought that other companies might be following agile/scrum methodologies for the Hadoop Development.

Also, I have one more question; How is stand-up meetings or client interaction process done in big Data projects ?

Re: How Hadoop works with agile and scrum methodologies? How does development, QA, Integration and Production release workflow involved in Hadoop projects ?

Super Collaborator

@Rakesh AN

I have worked for at least three companies trying to follow Agile/Scrum, and their development cycles of code does follow it.

It's hard to upgrade hundreds of Hadoop nodes and software versions, make sure they all work with other components of the cluster, all without breaking other pieces in two-week sprints, though.

Stand up meetings are all about perception management between team members and management. It again, has no special relationship or difference whether it is Hadoop development, web or mobile development, etc.