@Ravikiran Dasari: do you have any experience in shell scripts? Do you know how to write a bash script, or maybe a Python script?
Do you have a special scheduler that you use in your environment, or will you use cron? If you don't know, I guess it will become cron. Try if you are able to edit the crontab via entering this command on the shell
crontab -e
Either you get a list of cron jobs or an error message like 'You (<<userid>>) are not allowed to use this program (crontab)'
Now when you want to write a shell script the starting point is a simple text files, containing the commands you otherwise enter on the shell. The script file should start with a line aka 'shebang' providing the script interpreter to be used. I.e. on RedHat
- bash: #!/bin/bash
- python: #!/usr/bin/python
- perl: #!usr/bin/perl
If you decide to go for bash script, just create a file like this (you can use a different editor if you like):
vi ~/mycopyscript
enter in that script all your command
#!/bin/bash
dir = `date +%Y-%m-%d`
sftp ayosftpuser@IPaddredss << __MY_FTP_COMMANDS__
password
cd /sourcedir
get -Pr ${dir}
bye
__MY_FTP_COMMANDS__
#at this point the files should already be locally copied
hadoop fs -put -f ${dir} /destination
Save the script ( by entering <ESC>:wq in vi) Next make the script executable, and only allow access from the owner (you)
chmod 700 ~/mycopyscript
You should be able to execute it now:
~/mycopyscript
This script is just a starting point, and done plain simple, no error handling and no security, whoever reads the script also has access to the password, and no parameter (you must execute it at the date that the dir uses)
Still it should provide you with the basic idea of a shell script.
