I would strongly recommend upgrading from your older version of Kudu because there have been many improvements to address the issues you are describing. See the release notes for the Kudu releases after 1.3.0, many of these fixes will help you: https://kudu.apache.org/releases/1.4.0/docs/release_notes.html - The default size for Write Ahead Log (WAL) segments has been reduced from 64MB to 8MB. Additionally, in the case that all replicas of a tablet are fully up to date and data has been flushed from memory, servers will now retain only a single WAL segment rather than two. These changes are expected to reduce the average consumption of disk space on the configured WAL disk by 16x, as well as improve the startup speed of tablet servers by reducing the number and size of WAL segments that need to be re-read. - The Maintenance Manager has been improved to improve utilization of the configured maintenance threads. Previously, maintenance work would only be scheduled a maximum of 4 times per second, but now maintenance work will be scheduled immediately whenever any configured thread is available. This can improve the throughput of write-heavy workloads. - KUDU-2020 Fixed an issue where re-replication after a failure would proceed significantly slower than expected. This bug caused many tablets to be unnecessarily copied multiple times before successfully being considered re-replicated, resulting in significantly more network and IO bandwidth usage than expected. Mean time to recovery on clusters with large amounts of data is improved by up to 10x by this fix. https://kudu.apache.org/releases/1.6.0/docs/release_notes.html - Tablet server startup time has been improved significantly on servers containing large numbers of blocks. https://kudu.apache.org/releases/1.7.0/docs/release_notes.html - The strategy Kudu uses for automatically healing tablets which have lost a replica due to server or disk failures has been improved. The new re-replication strategy, or replica management scheme, first adds a replacement tablet replica before evicting the failed one. With the previous replica management scheme, the system first evicts the failed replica and then adds a replacement. The new replica management scheme allows for much faster recovery of tablets in scenarios where one tablet server goes down and then returns back shortly after 5 minutes or so. The new scheme also provides substantially better overall stability on clusters with frequent server failures. (see KUDU-1097). https://kudu.apache.org/releases/1.8.0/docs/release_notes.html - Introduced manual data rebalancer into the kudu CLI tool. The rebalancer can be used to redistribute table replicas among tablet servers. The rebalancer can be run via kudu cluster rebalance sub-command. Using the new tool, it’s possible to rebalance Kudu clusters of version 1.4.0 and newer. (note: CDH 5.16.1 doesn't include everything new from Kudu 1.8.0, only a few things like the rebalancer, but CDH 5.15.1 includes everything from Kudu 1.7.0 and earlier) If you can, upgrade to CDH 5.15.1 or CDH 5.16.1 There are also many other improvements unrelated to startup time that I have not called out here, such as greatly reducing the thread count, various optimizations, many other bug fixes, and lots of improvements for observability and operability.
... View more
