Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to sqoop automatically on daily basis.

Highlighted

How to sqoop automatically on daily basis.

New Contributor
 
4 REPLIES 4

Re: How to sqoop automatically on daily basis.

Please use oozie for scheduling sqoop a a given frequency : daily, hourly

https://oozie.apache.org/docs/4.0.0/DG_SqoopActionExtension.html

Re: How to sqoop automatically on daily basis.

@Gayathri Devi,

You can write a simple shell script which performs the sqoop import and run the shell script using a cronjob on a timely basis.

Below link gives different examples on how to run a cron job

http://www.thegeekstuff.com/2009/06/15-practical-crontab-examples

https://askubuntu.com/questions/2368/how-do-i-set-up-a-cron-job

Ex: You can put the shell script in /etc/cron.daily and run

crontab -e
0 1 * * * <path to script>   ----> run every day at 1 AM

Thanks,

Aditya

Re: How to sqoop automatically on daily basis.

@Gayathri Devi,

If you feel that cron is complex then I would suggest jenkins which most of the people use for CI . It has a nice UI where you have an option of doing periodic build. Just create a small job in jenkins. Add option "Build periodically" and give the cron expression to build each day. You can find cron expression easily by googling. And the build step would be only to call the script which you have written.

You can also use python/perl or any other scripting languages if you are not used to shell scripting. Even from jenkins or crontab your job is only to call the script which you have written.

Hope this helps :)

Thanks,

Aditya

Re: How to sqoop automatically on daily basis.

Mentor

@Gayathri Devi

You already have a script created sample.sh ,Now save it in some directory like /home/devi/sample.sh make sure the file is executable then the entry in the crontab should be like below

invoke the crontab in edit mode

# crontab -e 

Paste your code

0 1 * * * /home/devi/sample.sh ----> run every day at 1 AM

save the file

You can adjust the test by changing the time to every 2 minute

*/2 * * * * /home/devi/sample.sh ----> run every 2 minutes

There are many online sources that can help

Don't have an account?
Coming from Hortonworks? Activate your account here