- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Is there any relation of no of buckets with number of reducers? are both should be equal in hive?
- Labels:
-
Apache Hive
Created ‎03-22-2016 06:08 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎03-22-2016 01:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The command set hive.enforce.bucketing = true;
allows the correct number of reducers and the cluster by column to be automatically selected based on the table. Otherwise, you would need to set the number of reducers to be the same as the number of buckets.
Created ‎03-22-2016 11:40 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As per hive documentation what you have specified at the time of creation of bucket table with no.of buckets is become the number of reducers.If you have huge data single file contains different data.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL+BucketedTables
Created ‎03-22-2016 01:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@ thanks Kazan, there is bit correction with your ans,The command set hive.enforce.bucketing = true;
allows the correct number of reducers and the cluster by column to be automatically selected based on the table. Otherwise, you would need to set the number of reducers to be the same as the number of buckets.
Created ‎03-22-2016 01:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The command set hive.enforce.bucketing = true;
allows the correct number of reducers and the cluster by column to be automatically selected based on the table. Otherwise, you would need to set the number of reducers to be the same as the number of buckets.
