I was actually trying copy hdfs files from hive table which is multi partitioned. I was receiving old partition files which are giving difference in source table data and local path for most recent partitions. Example:
From the above files, there're 3 directories for 2020-06-29 partition and "/usr/hive/warehouse/sample_db.db/test_table/country=US/dt=2020-06-29/1236" is the latest file which is pointed for the partition. So, I was trying to fetch latest file for each partition before copying data from hdfs to s3.
I've tried the below command but it's giving only one latest file.