You can use a script like this to create snapshots of old and new files - i.e. search files which are older than 3 days and search for files which are newer than 3 days, just make sure, you use the correct path to the cloudera jars. In the case of CDH5.15:   #!/bin/bash now=`date +"%Y-%m-%dT%H:%M:%S"` hdfs dfs -rm /data/cleanup_report/part=older3days/* hdfs dfs -rm /data/cleanup_report/part=newer3days/* hadoop jar /opt/cloudera/parcels/CDH/jars/search-mr-1.0.0-cdh5.15.1.jar org.apache.solr.hadoop.HdfsFindTool -find /data -type d -mtime +3 | sed "s/^/${now}\tolder3days\t/" | hadoop fs -put - /data/cleanup_report/part=older3days/data.csv hadoop jar /opt/cloudera/parcels/CDH/jars/search-mr-1.0.0-cdh5.15.1.jar org.apache.solr.hadoop.HdfsFindTool -find /data -type d -mtime -3 | sed "s/^/${now}\tnewer3days\t/" | hadoop fs -put - /data/cleanup_report/part=newer3days/data.csv   Then create an external table with partitions on top of this HDFS folder. ... View more

Thanks for the reply ! Views are already created by joining many underlying table, hence joining the views again for data aggregation will result performance issue. Here are the two approach i came up with  1. Extract data from Hive view into files. 2. Create intermediate Hive tables and load data extracted from views. 3. Join the new hive tables to generate the final file.   Another approach to use PySpark to read data from views directly , aggreate and transform the data and generate the final output file. ... View more

Got it working on my cluster. Thanks Lars and Eric.   Cheers, Anand ... View more