question Need to bring prod hive table data into test environment using distcp. in Support Questions

Need to bring prod hive table data into test environment using distcp.

muthaiyaprabhu — Fri, 07 Apr 2017 00:31:04 GMT

I need to bring prod hive table data into test hive table. Since it's a hadoop to hadoop, i can't use sqoop, hence i can use discp to transfer data across the clusters. But i have one more scenario to be handled while bringing data, that is filtering. Say i have 10 million records in prod hive table, i want to filter using some criteria and bring it to test table. is there a way to give filter parameters in distcp command on the fly? Or any other suggestions? Thanks in advance.

Re: Need to bring prod hive table data into test environment using distcp.

namaheshwari — Fri, 07 Apr 2017 00:41:58 GMT

You can use distcp -filters to ignore few path, patterns

Refer this:

http://www.ericlin.me/how-to-use-filters-to-exclude-files-when-in-distcp

hadoop distcp -filters /path/to/filterfile.txt hdfs://source/path hdfs://destination/path

Re: Need to bring prod hive table data into test environment using distcp.

muthaiyaprabhu — Fri, 07 Apr 2017 01:51:52 GMT

Thanks Namit Maheshwari, data i am bringing into test is hive data, i need to filter using some criteria, like where condition in hive query. distcp -filters to exclude some files right, not on the data level. I want to filter the hive data using some criteria in production, and then want to bring the filtered data into test region.

Re: Need to bring prod hive table data into test environment using distcp.

balavignesh_nag — Fri, 07 Apr 2017 02:12:31 GMT

@Prabhu Muthaiyan

Filter the data from hive prod and load it into a file and then as mentioned by @Namit Maheshwari use distcp to transfer between different environments. If you want to limit the data without any filters being applied filter only a set of files under a HDFS folder.

Re: Need to bring prod hive table data into test environment using distcp.

muthaiyaprabhu — Fri, 07 Apr 2017 19:50:44 GMT

Thank you Bala!

Re: Need to bring prod hive table data into test environment using distcp.

balavignesh_nag — Fri, 07 Apr 2017 19:58:59 GMT

@Prabhu Muthaiyan Glad that it helped you. Happy Hadooping!!