Thanks youe reply.  Unfortunately, an upgrade is not available at this time in my company. I have rebalance my tablet server and modified the config 【 maintenance_manager_num_threads 】to 8 , 【 block_cache_capacity_mb 】 to 512MB, 【 memory_limit_hard_bytes 】to 60G。 AND then I try to run MR job on yarn, when get map counts with 96, and memory with 194G on yarn, kudu server is stable. So I continue to run a few job on yarn to to observe Kudu, and kudu server is still stable. So I think it is ok and set up scheduling tasks.   But when today, a job run 179 maps, the kudu server is random quit... this is the memory detail one of tablet server     ... View more

Thanks for your response. But I wanted to know all replication at once. I mean place of single table it should show like table name and replication factor.   Also, I left the project so unable to test the query.  ... View more

Hi,   As of Kudu v1.10 it's not possible to restrict access on a per-partition (i.e. per-tablet) basis.     Thanks,   Alexey ... View more

Thank you for the clarification,  Alexey. Much appreciated Edit: come to think of it: do you know anything about the relative efficiencies of upsert with ignoreNulls versus retrieving a dataframe from the table, doing my modifications in memory, and then upserting? Does kudu/spark do something similar under the hood, so there is little expected performance gain, or is it really a less "expensive" operation to do an update with ignoreNulls? 🙂 ... View more

Ah, there is another thread which might be helpful:    https://community.cloudera.com/t5/Support-Questions/Kudu-to-HDFS-data-load-timestamp-issue/td-p/93646  ... View more

Hi,   Kudu requires the machine clock of master and tablet servers nodes is synchronized using NTP : https://kudu.apache.org/docs/troubleshooting.html#ntp   Kudu is tested with ntpd, but I guess chronyd might work as well.  Whether using ntpd or chronyd, it's necessary to make sure the machine's clock is synchronized so ntp_adjtime() Linux system call doesn't return an error (see http://man7.org/linux/man-pages/man2/adjtimex.2.html for more technical details).   It's not enough just to have ntpd (or chronyd) running.  It's necessary to make sure the clock is synchronized.  I would verify that the NTP daemon is properly configured and tracks the clocks of the reference servers.  For the instructions to check the sync status of machine's clock, see https://kudu.apache.org/docs/troubleshooting.html#ntp if using ntpd or https://docs.fedoraproject.org/en-US/Fedora/18/html/System_Administrators_Guide/sect-Checking_if_chrony_is_synchronized.html for chronyd.   Hope this helps,   Alexey   ... View more

Hello, For anyone who has this problem. What I have observed is that in java/Talend, getTime() gives you a value in milliseconds. In Kudu, it expects the value to be in microseconds. So a factor of 1000. In Talend or java, after I get my date.getTime() value, I multiple it by 1000. Then in Kudu I get the correct date. I do not know if this is correct for everyone, but for me I get the desired result.   ... View more

Hi Alexey,   Thanks very much for your response.   What I mean by "runs a second command" is that if you execute something like this:   kudu cluster ksck masters.domain.com > output.txt   You will notice that the "Errors:" section at the very bottom of the command output isn't actually written to the file.  This holds true if you attempt to store the output in a variable.  This leads me to believe that part of the output isn't actually part of the original command execution.   I've found a way to work around this in shell programatically (if you could call it that).  I found the Linux "script" command, which captures output to screen, rather than just the output from a command itself.  I had to use the "-e -c" parameters to get it to function as a cron job for automated monitoring / alerting, but it appears to work as I desired.   Hopefully anyone else looking to perform automated monitoring/alerting of things not covered by Cloudera Manager for Kudu find this post helpful.   Regards,   Greg Frair ... View more

I would strongly recommend upgrading from your older version of Kudu because there have been many improvements to address the issues you are describing.   See the release notes for the Kudu releases after 1.3.0, many of these fixes will help you:   https://kudu.apache.org/releases/1.4.0/docs/release_notes.html  - The default size for Write Ahead Log (WAL) segments has been reduced from 64MB to 8MB. Additionally, in the case that all replicas of a tablet are fully up to date and data has been flushed from memory, servers will now retain only a single WAL segment rather than two. These changes are expected to reduce the average consumption of disk space on the configured WAL disk by 16x, as well as improve the startup speed of tablet servers by reducing the number and size of WAL segments that need to be re-read.  - The Maintenance Manager has been improved to improve utilization of the configured maintenance threads. Previously, maintenance work would only be scheduled a maximum of 4 times per second, but now maintenance work will be scheduled immediately whenever any configured thread is available. This can improve the throughput of write-heavy workloads.  - KUDU-2020 Fixed an issue where re-replication after a failure would proceed significantly slower than expected. This bug caused many tablets to be unnecessarily copied multiple times before successfully being considered re-replicated, resulting in significantly more network and IO bandwidth usage than expected. Mean time to recovery on clusters with large amounts of data is improved by up to 10x by this fix.   https://kudu.apache.org/releases/1.6.0/docs/release_notes.html  - Tablet server startup time has been improved significantly on servers containing large numbers of blocks.   https://kudu.apache.org/releases/1.7.0/docs/release_notes.html  - The strategy Kudu uses for automatically healing tablets which have lost a replica due to server or disk failures has been improved. The new re-replication strategy, or replica management scheme, first adds a replacement tablet replica before evicting the failed one. With the previous replica management scheme, the system first evicts the failed replica and then adds a replacement. The new replica management scheme allows for much faster recovery of tablets in scenarios where one tablet server goes down and then returns back shortly after 5 minutes or so. The new scheme also provides substantially better overall stability on clusters with frequent server failures. (see KUDU-1097).   https://kudu.apache.org/releases/1.8.0/docs/release_notes.html  - Introduced manual data rebalancer into the kudu CLI tool. The rebalancer can be used to redistribute table replicas among tablet servers. The rebalancer can be run via kudu cluster rebalance sub-command. Using the new tool, it’s possible to rebalance Kudu clusters of version 1.4.0 and newer.   (note: CDH 5.16.1 doesn't include everything new from Kudu 1.8.0, only a few things like the rebalancer, but CDH 5.15.1 includes everything from Kudu 1.7.0 and earlier)   If you can, upgrade to CDH 5.15.1 or CDH 5.16.1   There are also many other improvements unrelated to startup time that I have not called out here, such as greatly reducing the thread count, various optimizations, many other bug fixes, and lots of improvements for observability and operability. ... View more

thanks for these informations, a little question : where must I define the custom mapping ?   Best regards   Christophe ... View more

@Andreyeff Another thing you can try doing is increasing the raft heartbeat interval from 500ms to 1500ms or even 3000ms, see https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_raft_heartbeat_interval_ms   This will affect your recovery time by a few seconds if a leader fails since by default, elections don't happen for 3 missed heartbeat periods (controlled by https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_leader_failure_max_missed_heartbeat_periods ) ... View more

Right, to make the fix into Impala it's necessary to relink impalad with patched Kudu client.   impalad is linked against the kudu_client dynamically, so in theory it might be possible just to replace the libkudu_client.so.0 library with the patched version.  However, that's really messy and I would not recommend that.  If you use CDH anyway, the best option is to wait for next release of CDH.  I don't know what version that will be, though.   If you want a workaround, set the -- authn_token_validity_seconds flag to be months or even one year long  (i.e.  -- authn_token_validity_seconds= 31536000) and restart Kudu masters.  You will need to enable experimental flags as well (i.e. add  --unlock_experimental_flags). ... View more

Just for better visibility I wanted to refer to corresponding thread at user@kudu.apache.org:   https://lists.apache.org/thread.html/99bb508aaf2a33d823066ddeb7ba8b58e14eb577c5a89f56359f5cd4@%3Cuser.kudu.apache.org%3E   Also, there were responses at #kudu-general Slack channel at https://getkudu.slack.com/   https://getkudu.slack.com/archives/C0CPXJ3CH/p1523339652000175 ... View more

Thank you   Alexey, the port in case 1 was endeed the problem. I don't know where I got the 7077, I guess **bleep**ching back and forth form Kudu to Spark addresses got them mixed up. In case 2, I am working the same machine but strangely, changing the ip from localhost to 172.17.0.43 made it work. Thanks again! ... View more

@Alexey1c: Thanks for your reply. The string before the “.” Is in fact the database name. I contected Cloudera Support and they confirmed that the CREATE EXTERNAL TABLE statement on the Kudu UI is erroneous. ... View more

Cloudera publishes Kudu binary packages in accordance with the internal release cycle.  Per company's policy, we don't promise any particular dates for features/builds availability.   However, there are binary packages of Kudu 1.3.0 in the Cloudera's repository.  E ven though our binary says 1.3.0 it may include backports that were included in the upstream 1.3.1.   When it comes to bugfix patches, we often include backports that aren't in the exact upstream release.  You can find package repos URLs at the https://www.cloudera.com/documentation/kudu/latest/topics/kudu_installation.html page.  It might happen that the patch you are looking for is in cloudera's package v1.3.0 already.  For details, please take a look at the release notes: https://www.cloudera.com/documentation/kudu/latest/topics/release_notes.html#relnotes_1_3_0   Is there any specific feature/patch that you are expecting with 1.3.1 release?   ... View more

By default, CM will warn when 50% of a process' FDs are in use. Also by default, Kudu's block manager system will use 50% of the FDs available to the process. So, after accounting for some additional FDs for WALs, Kudu ends up using a little over 50% of the available FDs and CM warns about it.   If this bothers you, you can: Reconfigure Kudu's block_manager_max_open_files to some fixed value below 16384. The default value of -1 means Kudu will use 50% of what's available (16384 in your case). Reconfigure CM to warn at a higher threshold than 50%. Wait for CDH 5.11, where Kudu's percent usage was dropped from 50% to 40%.   ... View more