question Re: specifying the number of reducers for Phoenix bulk load tool in Archives of Support Questions (Read Only)

specifying the number of reducers for Phoenix bulk load tool

arunpoy — Thu, 10 Nov 2016 18:11:02 GMT

IS there a way to specify the number of reducers for the phoenix CSV bulk load utility that uses the Mapreduce method to load data into hbase.

Re: specifying the number of reducers for Phoenix bulk load tool

ssoldatov — Thu, 10 Nov 2016 18:21:18 GMT

MR job creates 1 reducer per region. So if you loading data to an empty table you may presplit table from HBase shell or use salting during table creation.

Re: specifying the number of reducers for Phoenix bulk load tool

arunpoy — Thu, 10 Nov 2016 18:33:02 GMT

hi @ssoldatov, Thanks for your reply. So it wont honour the command line argument if i prvide like this

hadoop jar phoenix-4.8.1-HBase-1.1-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool -Dmapreduce.job.reduces=4 (not the full command)

though i specified 4 reducers, it just considered only.

Re: specifying the number of reducers for Phoenix bulk load tool

ssoldatov — Thu, 10 Nov 2016 18:56:33 GMT

No. During the start of the MR job you may see message like:

mapreduce.MultiHfileOutputFormat: Configuring 20 reduce partitions to match current region count

That's exactly the number of reducers that will be created. How many of them will be running in parallel depends on the MR engine configuration.