About mpercy

mpercy · ‎05-15-2018

I looked into this a little more today, out of interest, and added some additional info about how to make it work on EL6 here: https://issues.apache.org/jira/browse/KUDU-2442 Regardless, if you are using a parcel install, you must first symlink the include files and the libraries into /usr/ or /usr/local, something like this: sudo ln -s /opt/cloudera/parcels/CDH/include/kudu /usr/local/include/ sudo ln -s /opt/cloudera/parcels/CDH/lib64/libkudu_client.so /usr/local/lib64/ sudo ln -s /opt/cloudera/parcels/CDH/lib64/libkudu_client.so.0 /usr/local/lib64/ sudo ln -s /opt/cloudera/parcels/CDH/lib64/libkudu_client.so.0.1.0 /usr/local/lib64/

mpercy · ‎05-15-2018

I looked at this a little more and I think kudu-python install on RHEL 6 / Centos 6 is broken because of the lack of Python 2.6 support in newer versions of pip. There are a couple of related JIRAs on this issue: https://issues.apache.org/jira/browse/KUDU-1705 and https://issues.apache.org/jira/browse/KUDU-2442 Another issue is that the parcel does not link the include files and libraries into a place that pip can find them. You may be able to work around that with putting symlinks from /opt/cloudera/parcels/CDH/includes/kudu into /usr/include/kudu and /opt/cloudera/parcels/CDH/lib64/*kudu* into /usr/lib64/ to see if that helps solve the problem on a newer OS version.

mpercy · ‎05-15-2018

Hi elkarel, This is a very old thread and has some outdated information and expired links. Please include any problems you are facing including any error messages you are seeing, your version, platform, etc. Regarding your question about the documentation, there is no longer any special procedure to install Kudu. With the latest versions of CDH, Kudu is included in the parcel: https://www.cloudera.com/documentation/enterprise/5-14-x/topics/kudu_install_cm.html Thanks, Mike

mpercy · ‎05-09-2018

Kudu does not use HDFS at all. It requires its own storage space. If you use 3x replication (the default) and no compression then Kudu will take 3x the amount of space that you ingest. However Kudu tends to efficiently encode and compress data so you will have to evaluate how much space Kudu takes based on the schema and data ingestion patterns you have. The more RAM you give Kudu the better it will perform... treat Kudu like a database (think MySQL or Vertica). Right now there is no way to specify a quota, the only available settings related to that are: --fs_wal_dir_reserved_bytes ( https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_fs_wal_dir_reserved_bytes ) and --fs_data_dirs_reserved_bytes ( https://kudu.apache.org/docs/configuration_reference.html#kudu-tserver_fs_data_dirs_reserved_bytes ) If you need to closely control the amount of space Kudu uses then you can consider putting it on its own partitions or machines. However if it possible to put Kudu on the same machines that have HDFS running on them if you want to do that. Hope that helps!

mpercy · ‎05-07-2018

Unfortunately there is no support for storing data on ADLS at this time. Regarding backup and restore, we are in the process of designing a backup solution into Kudu, however there is no ETA at this time for when it will be ready to use. You can follow the progress at https://issues.apache.org/jira/browse/KUDU-1575

mpercy · ‎06-28-2017

Impala heavily relies on parallelism for throughput so if you have 60 partitions for Kudu and 1800 partitions for Parquet then due to Impala's current single-thread-per-partition limitation you have built in a huge disadvantage for Kudu in this comparison. Please let us know if you re-run your comparison test.

mpercy · ‎06-27-2017

If you are under the scale limits consider increasing # of partitions. Impala tends to use one thread per partition when scanning.

mpercy · ‎06-27-2017

Could you check whether you are under the current scale recommendations for Kudu? We are working hard on increasing these limits and will try to do so for each coming release. Current scale limits for CDH 5.11 (Kudu 1.3): https://www.cloudera.com/documentation/kudu/latest/topics/kudu_known_issues.html#concept_cws_n4n_5z

mpercy · ‎06-27-2017

Can you also share how you partitioned your Kudu table?

mpercy · ‎06-27-2017

Make sure you run COMPUTE STATS after loading the data so that Impala knows how to join the Kudu tables. What is the total size of your data set? I am surprised at the difference in your numbers and I think they should be closer if tuned correctly. Regardless, if you don't need to be able to do online inserts and updates, then Kudu won't buy you much over the raw scan speed of an immutable on-disk format like Impala + Parquet on HDFS.

Online	Offline
Last Visited	‎05-09-2019 02:10 AM

Member Since	‎04-08-2014 11:43 PM
Last Visited	‎05-09-2019 02:10 AM
Posts	70
Kudos received	19

Cloudera Community

Re: Apache kudu

Re: Apache kudu

Re: Apache kudu

Re: Apache kudu

Re: Apache kudu

Re: Using kudu with Python

Re: Using kudu with Python

Re: Using kudu with Python

Re: Does Kudu need additional resources appart fro...

Re: Kudu file storage on ADLS

Re: kudu is slower than parquet?

Re: kudu is slower than parquet?

Re: kudu is slower than parquet?

Re: kudu is slower than parquet?

Re: kudu is slower than parquet?