Simple example of Jenkins-HDP integration

[#1]

First of all, you need a Hadoop user with correct permission with Ranger (I will use mine : mlanciau), some HDFS quota (if configured), and some YARN queue (I hope it is configured)

For this example, I am using default Jenkins configuration with SSH plugin and the TeraSort benchmark suite.

1. Configure SSH remote hosts in Jenkins by entering your credential (of course you can use different config)

2. Create a new item (on the main portal), I called it Hadoop performance, then click on it and configure

3. Click on "This project is parameterised" and add these 2 parameters (you may want to change the parameter ;-))

4. Go to build and click on add a step to the build and choose Execute shell script on remote host using ssh

5. Then use below information to configure the different scripts (you may want to configure your own user)

1. hdfs dfs -rm -r -skipTrash /user/mlanciau/teragen ; echo 'deleting directory'
2. hdfs dfs -rm -r -skipTrash /user/mlanciau/terasort ; echo 'deleting directory'
3. hdfs dfs -rm -r -skipTrash /user/mlanciau/teravalidate ; echo 'deleting directory'
4. hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teragen -Dmapred.map.tasks=${mapperNbr} 10000 /user/mlanciau/teragen
5. hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar terasort -Dmapred.reduce.tasks=${reducerNbr} /user/mlanciau/teragen /user/mlanciau/terasort
6. hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teravalidate /user/mlanciau/terasort /user/mlanciau/teravalidate

6. Click on save

7. Click on Build with Parameters, check the parameter values and then Build

8. Follow the progress by clicking on the step in progress and Console Output !

If you want you can also check execution time, success / failure thanks to the Jenkins UI

Cheers 😉

[#2]

Cool stuff

[#3]

That is amazing!