I'm terribly sorry for the necro-bump. The issue I'm experiencing seems to be the same one the OP was facing. We're running CDH 6.3, and I encountered a problem when trying to install the Kudu Python client. I created the symlinks as recommended by mpercy, but I'm still unable to install the kudu-python client. Our cluster does not have direct access to the internet, so when possible we use an offline install. The official documentation says that the  Kudu C++ client libraries and headers are needed for the Kudu Python client. On Oracle Linux 7 trying to install devtoolset-3-toolchain ends in a failure, as a number of dependencies are missing:   Error: Package: devtoolset-3-gdb-7.8.2-38.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libpython2.6.so.1.0()(64bit) Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libmpfr.so.1()(64bit) Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libgmp.so.3()(64bit) Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libgmp.so.3()(64bit) Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libmpfr.so.1()(64bit) Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libgmp.so.3()(64bit) Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64) Requires: libmpfr.so.1()(64bit) Disregarding that, running pip install --no-index --find-links file:///data0/home/jkovacs/kudu-python-1.10.0.tar.gz  results in the following errors: ERROR: Command errored out with exit status 1: command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rK2Qbk/kudu-python/pip-egg-info cwd: /tmp/pip-install-rK2Qbk/kudu-python/ Complete output (43 lines): Building from system prefix /usr/local /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/client.pxd tree = Parsing.p_module(s, pxd, full_module_name) /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/schema.pxd tree = Parsing.p_module(s, pxd, full_module_name) /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/errors.pxd tree = Parsing.p_module(s, pxd, full_module_name) Compiling kudu/client.pyx because it depends on kudu/config.pxi. Compiling kudu/errors.pyx because it depends on /usr/lib64/python2.7/site-packages/Cython/Includes/libcpp/string.pxd. Compiling kudu/schema.pyx because it depends on kudu/config.pxi. [1/3] Cythonizing kudu/client.pyx [2/3] Cythonizing kudu/schema.pyx [3/3] Cythonizing kudu/errors.pyx WARNING: The wheel package is not available. DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at <a href="<a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a>" target="_blank"><a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a</a>> WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3910>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3110>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3090>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e50>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e90>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/ ERROR: Could not find a version that satisfies the requirement pytest-runner (from versions: none) ERROR: No matching distribution found for pytest-runner Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-rK2Qbk/kudu-python/setup.py", line 216, in <module> test_suite="kudu.tests" File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup _install_setup_requires(attrs) File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs replace_conflicting=True, File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve replace_conflicting=replace_conflicting File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match return self.obtain(req, installer) File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain return installer(requirement) File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg return fetch_build_egg(self, req) File "/usr/lib/python2.7/site-packages/setuptools/installer.py", line 121, in fetch_build_egg raise DistutilsError(str(e)) distutils.errors.DistutilsError: Command '['/usr/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp8eqkO2', '--quiet', 'pytest-runner']' returned non-zero exit status 1 ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. I'd like to ask for some guidance, what to try next?     Thank you!   ... View more

@xuehan2216 @sdutta007    You can perform CTAS - please refer the below reference  CREATE TABLE new_table PRIMARY KEY (ts, name) PARTITION BY HASH(name) PARTITIONS 8 STORED AS KUDU AS SELECT ts, name, value FROM old_table; ... View more

You need to download/distribute/activate the KUDU parcel in addition to the CDH parcel. First, add http://archive.cloudera.com/kudu/parcels/5.11.0/ to your parcel repository list in Cloudera Manager. Then a KUDU parcel should show up in the list of available parcels. ... View more

Based on the error message, it sounds like the Cloudera Manager server timed out while trying to access the parcel repo URL you specified. Perhaps double check that you didn't introduce any typos, and that you can download <parcel repo URL>/manifest.json using curl or wget from the machine running the CM server? Since this is an issue with CM and parcel installation, if you can't solve the problem I suggest you use this forum: http://community.cloudera.com/t5/Cloudera-Manager-Installation/bd-p/CMInstall ... View more

there is a uuid function in impala that you can use to generate surrogate keys for kudu. or you can write an impala udf to generate unique bigints. ... View more

@sdutta007   the following key words are supported while creating kutu table     PARTITIONED STORED AS LOCATION ROWFORMAT     could you follow this syntax and let me know if you need more info. you can perform a HASH partition or Range partition. or mix  using Distributed by Range or DISTRIBUTE BY HASH (id ) INTO 4 BUCKETS,   CREATE TABLE `TableName` ( `id` BIGINT, `name` STRING ) DISTRIBUTE BY RANGE(id) or DISTRIBUTE BY HASH (id) INTO 16 BUCKETS TBLPROPERTIES ( 'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler', 'kudu.table_name' = 'Your Table Name', 'kudu.master_addresses' = 'kudu-master-hostname:7051', 'kudu.key_columns' = 'id' );   ... View more

Those warnings are likely harmless. Is there any actual failure during the upserts? ... View more

The "rolling extension" process you're suggesting looks reasonable. It should serve as a workaround for Kudu's current inability to add new data directories. ... View more

You could use 'tinker step 500' and have the effect that stepping would only be enabled for time differences more than 500ms. I wouldn't consider this breaking your production environment, but I guess you may have some reason that '-x' is important to you. We'll work on addressing this in a future release so that no system-wide changes are necessary. -Todd ... View more

Not sure if you've resolved your problem or not, but as a general note, you're using a very old version of Impala, and that version may have interoperability problems with a modern Kudu.   It'd be best if you switched to Kudu and Impala from CDH 5.10 or newer. Beginning with CDH 5.10 the special "Impala-Kudu" release of Impala no longer exists as regular CDH Impala can interoperate with Kudu. ... View more

It's also possible to range partition on multiple columns, so you could potentially skip the hash partitions if you wanted (although it requires specifying many more explicit range partitions):   CREATE TABLE sample_table (year INT,month INT,day INT,some_id INT,some_value INT, PRIMARY KEY(year,month,day,some_id)) RANGE(year, month) ( PARTITION VALUE = (2017, 01), PARTITION VALUE = (2017, 02), PARTITION VALUE = (2017, 03), PARTITION VALUE = (2017, 04), PARTITION VALUE = (2017, 05), PARTITION VALUE = (2017, 06), PARTITION VALUE = (2017, 07), PARTITION VALUE = (2017, 08), PARTITION VALUE = (2017, 09), PARTITION VALUE = (2017, 10), PARTITION VALUE = (2017, 11), PARTITION VALUE = (2017, 12), PARTITION VALUE = (2018, 01), PARTITION VALUE = (2018, 02), ) STORED AS KUDU;   I think the reason you had to specify the year and month up front in the column list is that currently Kudu tables have a restriction that the primary key columns must come first in the column list. ... View more

Hi @Tim Armstrong and @Adar   I am able to resolve github ssl certificate issue.   I am trying to download quickstart ova file from below link but it seems like this repo is very slow. Is there any other repo from where we can download this kudu quickstart vm file.   http://cloudera-kudu-beta.s2.amazonaws.com/cloudera-quickstart-vm-5.10.0-kudu-virtualbox.ova   Thanks Rahul ... View more

Just to close this thread.  After a bit more investigation, we found that despite the error message received the table was actually populated with the data.   Thanks for your help. ... View more

Thank you! It's really because storm workers are suffering GC pauses! We will check the code of storm workers. ... View more

By default, CM will warn when 50% of a process' FDs are in use. Also by default, Kudu's block manager system will use 50% of the FDs available to the process. So, after accounting for some additional FDs for WALs, Kudu ends up using a little over 50% of the available FDs and CM warns about it.   If this bothers you, you can: Reconfigure Kudu's block_manager_max_open_files to some fixed value below 16384. The default value of -1 means Kudu will use 50% of what's available (16384 in your case). Reconfigure CM to warn at a higher threshold than 50%. Wait for CDH 5.11, where Kudu's percent usage was dropped from 50% to 40%.   ... View more