Created 02-21-2018 04:38 AM
Hi,
I'm trying to implement Hadoop project and i'm researching on how the SDLC workflow involved in the Hadoop project.
Thanks,
Created 02-21-2018 06:49 PM
I don't believe hardware or infrastructure setup should follow any such workflow.
Maybe you need separate environments to isolate workloads, but other than that, it's the code itself that follows development patterns as anything else.
Examples of such code could include MapReduce, Hive scripts, Oozie jobs, Spark processes, NiFi dataflows, etc.
In terms of MapReduce or Spark, you can use CI/CD processes to build code and push it to HDFS, and submit it to YARN to run once, or submit it to Oozie to run on a schedule.
Hadoop itself just offers HDFS, YARN, and MapReduce. It's everything else that is very specific to your needs and processes.
Created 02-21-2018 06:49 PM
I don't believe hardware or infrastructure setup should follow any such workflow.
Maybe you need separate environments to isolate workloads, but other than that, it's the code itself that follows development patterns as anything else.
Examples of such code could include MapReduce, Hive scripts, Oozie jobs, Spark processes, NiFi dataflows, etc.
In terms of MapReduce or Spark, you can use CI/CD processes to build code and push it to HDFS, and submit it to YARN to run once, or submit it to Oozie to run on a schedule.
Hadoop itself just offers HDFS, YARN, and MapReduce. It's everything else that is very specific to your needs and processes.
Created 02-22-2018 04:34 AM
Thanks for the update !!!
We are also working in the same fashion as you said, but I thought that other companies might be following agile/scrum methodologies for the Hadoop Development.
Also, I have one more question; How is stand-up meetings or client interaction process done in big Data projects ?
Created 02-22-2018 08:19 PM
I have worked for at least three companies trying to follow Agile/Scrum, and their development cycles of code does follow it.
It's hard to upgrade hundreds of Hadoop nodes and software versions, make sure they all work with other components of the cluster, all without breaking other pieces in two-week sprints, though.
Stand up meetings are all about perception management between team members and management. It again, has no special relationship or difference whether it is Hadoop development, web or mobile development, etc.