https://community.cloudera.com/t5/Archives-of-Support-Questions/How-GIT-Jenkins-are-related-to-Hadoop-Spark-jobs/m-p/124207#M47277 <P>These tools are used similarly with any software SDLC, just you will be developing software being executed on a Hadoop/Spark cluster. You can still build your jars the same way and use GIT as your source code repository. You will be submitting the job for execution in a distributed cluster. However, there are pseudo clusters for development. For example you can use hadoop mini cluster: <A href="https://github.com/sakserv/hadoop-mini-clusters" target="_blank">https://github.com/sakserv/hadoop-mini-clusters</A> </P><P>A good reference on how to use this mini cluster for testing: <A href="http://www.lopakalogic.com/articles/hadoop-articles/hadoop-testing-with-minicluster/" target="_blank">http://www.lopakalogic.com/articles/hadoop-articles/hadoop-testing-with-minicluster/</A></P><P>For Spark development you could use Spark standalone.</P> Thu, 29 Dec 2016 08:16:43 GMT cstanca 2016-12-29T08:16:43Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-GIT-Jenkins-are-related-to-Hadoop-Spark-jobs/m-p/124206#M47276 <P>Hi there,</P><P>I know Jenkins &amp; GIt in general. But, I'm not aware of how Jenkins/GIT plays role in Hadoop projects..</P><P>Please let me know your information on this. Thanks in advance.</P><P>Regards,</P><P>Jee </P> Mon, 28 Nov 2016 11:17:19 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-GIT-Jenkins-are-related-to-Hadoop-Spark-jobs/m-p/124206#M47276 JeevaKrish 2016-11-28T11:17:19Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-GIT-Jenkins-are-related-to-Hadoop-Spark-jobs/m-p/124207#M47277 <P>These tools are used similarly with any software SDLC, just you will be developing software being executed on a Hadoop/Spark cluster. You can still build your jars the same way and use GIT as your source code repository. You will be submitting the job for execution in a distributed cluster. However, there are pseudo clusters for development. For example you can use hadoop mini cluster: <A href="https://github.com/sakserv/hadoop-mini-clusters" target="_blank">https://github.com/sakserv/hadoop-mini-clusters</A> </P><P>A good reference on how to use this mini cluster for testing: <A href="http://www.lopakalogic.com/articles/hadoop-articles/hadoop-testing-with-minicluster/" target="_blank">http://www.lopakalogic.com/articles/hadoop-articles/hadoop-testing-with-minicluster/</A></P><P>For Spark development you could use Spark standalone.</P> Thu, 29 Dec 2016 08:16:43 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-GIT-Jenkins-are-related-to-Hadoop-Spark-jobs/m-p/124207#M47277 cstanca 2016-12-29T08:16:43Z