Support Questions
Find answers, ask questions, and share your expertise

specifying the number of reducers for Phoenix bulk load tool

IS there a way to specify the number of reducers for the phoenix CSV bulk load utility that uses the Mapreduce method to load data into hbase.

1 ACCEPTED SOLUTION

Super Collaborator

No. During the start of the MR job you may see message like:

mapreduce.MultiHfileOutputFormat: Configuring 20 reduce partitions to match current region count

That's exactly the number of reducers that will be created. How many of them will be running in parallel depends on the MR engine configuration.

View solution in original post

3 REPLIES 3

Super Collaborator

MR job creates 1 reducer per region. So if you loading data to an empty table you may presplit table from HBase shell or use salting during table creation.

hi @ssoldatov, Thanks for your reply. So it wont honour the command line argument if i prvide like this

hadoop jar phoenix-4.8.1-HBase-1.1-client.jar org.apache.phoenix.mapreduce.CsvBulkLoadTool -Dmapreduce.job.reduces=4 (not the full command)

though i specified 4 reducers, it just considered only.

Super Collaborator

No. During the start of the MR job you may see message like:

mapreduce.MultiHfileOutputFormat: Configuring 20 reduce partitions to match current region count

That's exactly the number of reducers that will be created. How many of them will be running in parallel depends on the MR engine configuration.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.