Generic HDFS data and Hive Database transfer autom...
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.
Exporting and importing data between different layers of environment like production, QA and development is a recurring task.
Due to security considerations, this environments cannot talk to each other. Hence we are using Amazon S3 storage as an intermediate storage point for transferring data seamlessly across environments. The automation of this task is expected to save close to 4 hours of manual intervention per occurrence.
The code can be re-used for disaster recovery automation.