I need to move the files from File Landing Area to HDFS on frequency basis. Planning to write a shell script to do this task? Is this better way of handling and invoking the shell from Oozie scheduler.
If so, could you please provide the list of sequence steps + configurations which are required to achieve the functionality.
A shell action would be the only method to move a file into HDFS. A few things to keep in mind:
1. The shell script could run on any NodeManager, so the file being copied has to be available from any NodeManager.
2. Configs in /etc/hadoop/conf must be correct on all NodeManagers so that the "hadoop fs" command will run with correct config.
Hope this helps.
Many thanks for your swift response. Could you please provide the list of steps to be configured (e.g. like job.properties,workflow) with an example it will be great.
No need to write the shell script.