Support Questions
Find answers, ask questions, and share your expertise

Hive BDR replication with more than 1000 tables

Hive BDR replication with more than 1000 tables

Explorer

HIVE replication limitations -

Hive replication has the following limitations:
  • Maximum number of databases: 100
  • Maximum number of tables per database: 1,000
  • Maximum number of partitions per table: 10,000. 
  • Maximum total number of tables (across all databases): 10,000
  • Maximum total number of partitions (across all tables): 100,000
  • Maximum number of indexes per table: 100


CDH 5.15.1
In webex, we noticed there are two jobs .
a) One job with 120 databases
b) another job with 1 database having 4386 tables

We randomly checked many tables to check the partitions but no partitions are created for randomly checked tables. 

Both BDR jobs are failing and customer has data around 300TB in both jobs

 

What is the solution in this case?