Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive BDR replication with more than 1000 tables

Highlighted

Hive BDR replication with more than 1000 tables

Explorer

HIVE replication limitations -

Hive replication has the following limitations:
  • Maximum number of databases: 100
  • Maximum number of tables per database: 1,000
  • Maximum number of partitions per table: 10,000. 
  • Maximum total number of tables (across all databases): 10,000
  • Maximum total number of partitions (across all tables): 100,000
  • Maximum number of indexes per table: 100


CDH 5.15.1
In webex, we noticed there are two jobs .
a) One job with 120 databases
b) another job with 1 database having 4386 tables

We randomly checked many tables to check the partitions but no partitions are created for randomly checked tables. 

Both BDR jobs are failing and customer has data around 300TB in both jobs

 

What is the solution in this case?

Don't have an account?
Coming from Hortonworks? Activate your account here