question Re: copy files within hdfs based on the modified time or access time in Support Questions

copy files within hdfs based on the modified time or access time

klksrinivas — Thu, 14 Jul 2016 17:28:21 GMT

I have to write a script to move files(csv) from one location in hdfs to another staging location in hdfs.(based on date) As of now I have to move files from April 2nd 2016. Later I have to schedule it so that files will be picked up for every 1 hr and moved to staging location. Hive tables are created on top of this staging location.

Re: copy files within hdfs based on the modified time or access time

ssingla — Fri, 15 Jul 2016 01:51:35 GMT

1) For moving files from 2nd april to another folder in hdfs.

for i in `hdfs dfs -ls /old_data/dataset/|grep "2016-04-02"|awk '{print $8}'`;do echo ${i}; hdfs dfs -mv ${i} /old_data/dataset/TEST/;done

2) Once the above is done you can just setup a crontab.

Please try this scenario out on a test folder in non prod.

Re: copy files within hdfs based on the modified time or access time

hrongali — Fri, 15 Jul 2016 02:28:11 GMT

@Krishna Srinivas

Have you tried the Falcon mirroring feature ? Instead of cluster to cluster replication, you can try replicating to different directories in the same cluster.

http://hortonworks.com/hadoop-tutorial/mirroring-datasets-between-hadoop-clusters-with-apache-falcon/

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_data_governance/content/section_mirroring_data_falcon.html

https://falcon.apache.org/HDFSDR.html