Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Contributor

First of all, you need a Hadoop user with correct permission with Ranger (I will use mine : mlanciau), some HDFS quota (if configured), and some YARN queue (I hope it is configured)

For this example, I am using default Jenkins configuration with SSH plugin and the TeraSort benchmark suite.

1. Configure SSH remote hosts in Jenkins by entering your credential (of course you can use different config)

5055-screenshot-2016-06-16-135831.png

2. Create a new item (on the main portal), I called it Hadoop performance, then click on it and configure

3. Click on "This project is parameterised" and add these 2 parameters (you may want to change the parameter ;-))

5108-screenshot-2016-06-20-045603.png

4. Go to build and click on add a step to the build and choose Execute shell script on remote host using ssh

5109-screenshot-2016-06-20-045734.png

5. Then use below information to configure the different scripts (you may want to configure your own user)

  1. hdfs dfs -rm -r -skipTrash /user/mlanciau/teragen ; echo 'deleting directory'
  2. hdfs dfs -rm -r -skipTrash /user/mlanciau/terasort ; echo 'deleting directory'
  3. hdfs dfs -rm -r -skipTrash /user/mlanciau/teravalidate ; echo 'deleting directory'
  4. hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teragen -Dmapred.map.tasks=${mapperNbr} 10000 /user/mlanciau/teragen
  5. hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar terasort -Dmapred.reduce.tasks=${reducerNbr} /user/mlanciau/teragen /user/mlanciau/terasort
  6. hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teravalidate /user/mlanciau/terasort /user/mlanciau/teravalidate

6. Click on save

7. Click on Build with Parameters, check the parameter values and then Build

8. Follow the progress by clicking on the step in progress and Console Output !

5060-screenshot-2016-06-16-141253.png

If you want you can also check execution time, success / failure thanks to the Jenkins UI

5064-screenshot-2016-06-16-144534.png

Cheers


screenshot-2016-06-16-140708.pngscreenshot-2016-06-16-140441.pngscreenshot-2016-06-16-140822.pngscreenshot-2016-06-16-140111.png
6,005 Views
Comments
Contributor

Cool stuff

That is amazing!

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 12:01 PM
Updated by:
 
Contributors
Top Kudoed Authors