@SBandaru - Below is an excellent article on HCC explaining distcp with Snapshots:
https://community.hortonworks.com/articles/71775/managing-hadoop-dr-with-distcp-and-snapshots.html
From the article:
- Source must support 'snapshots'
hdfs dfsadmin -allowSnapshot <path>
- Target is "read-only"
- Target, after initial baseline 'distcp' sync needs to support snapshots.
Process
- Identify the source and target 'parent' directory
- Do not initially create the destination directory, allow the first distcp to do that. For example: If I want to sync source `/data/a` with `/data/a_target`, do *NOT* pre-create the 'a_target' directory.
- Allow snapshots on the source directory
hdfs dfsadmin -allowSnapshot /data/a
- Create a Snapshot of /data/a
hdfs dfs -createSnapshot /data/a s1
- Distcp the baseline copy (from the atomic snapshot). Note: /data/a_target does NOT exists prior to the following command.
hadoop distcp /data/a/.snapshot/s1 /data/a_target
- Allow snapshots on the newly create target directory
hdfs dfsadmin -allowSnapshot /data/a_target
- At this point /data/a_target should be considered "read-only". Do NOT make any changes to the content here.
- Create a matching snapshot in /data/a_target that matches the name of the snapshot used to build the baseline
hdfs dfs -createSnapshot /data/a_target s1
- Add some content to the source directory /data/a. Make changes, add, deletes, etc. that need to be replicated to /data/a_target.
- Take a new snapshot of /data/a
hdfs dfs -createSnapshot /data/a s2
- Just for fun, check on whats changed between the two snapshots
hdfs snapshotDiff /data/a s1 s2
- Ok, now let's migrate the changes to /data/a_target
hadoop distcp -diff s1 s2 -update /data/a /data/a_target
- When that's completed, finish the cycle by creating a matching snapshot on /data/a_target
hdfs dfs -createSnapshot /data/a_target s2
That's it. You've completed the cycle. Rinse and repeat.
