Using kudu with Python

[#1]

Hi,

I am unable to use kudu with Python.

[cloudera@quickstart ~]$ sudo pip install kudu-python
Collecting kudu-python
Using cached kudu-python-1.2.0.tar.gz
Complete output from command python setup.py egg_info:
Cannot find installed kudu client.

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-GE6kA6/kudu-python/

[cloudera@quickstart ~]$ kudu -version
kudu 1.2.0-cdh5.10.2
revision f8f159f3c55f72ef026776f6176855e4f6e70271
build type RELEASE
built by jenkins at 07 Jun 2017 01:01:28 PST on kudu-centos66-19fa.vpc.cloudera.com
build id 2017-06-07_00-28-13

[cloudera@quickstart ~]$ python -V
Python 2.7.13 :: Anaconda 4.4.0 (64-bit)

This is what I am following:

https://www.cloudera.com/documentation/kudu/latest/topics/kudu_development.html

Any help?

[#2]

Thank you very much Mike for the follow-up!

Our client wanted us to stop using python, so I cannot verify your solution.

The system is CentOS 7.4.1708. 

I will return to this Jira issue in future if there is a need to continue with python-kudu.

Best Regards

[#3]

With Centos 7 I think the only thing required to make "pip install kudu-python" work out-of-the-box is to add the symlinks I mentioned. I'm going to work on getting those added by default as part of the parcel install in a future release.

[#4]

Nope, that "pip install kudu-python" certainly didn't work for me, claiming
"Cannot find installed kudu client.".
Oviously, kudu was accessible from CLI, so I guess it searches for some
directories that don't get created in parcel install.

Thanks

[#5]

Yes -- I saw the same behavior, and the workaround is adding the symlinks I detailed in a previous comment in this thread.

[#6]

Is there a chance a connect to multiple masters to allow Python client choose the needed leader itself?

Current connection string

# Connect to Kudu master server
client = kudu.connect(host='kudu.master', port=7051)

may prevent some automation tasks, because I have to choose the right leader first.

[#7]

You can connect to multiple masters by providing a list of master addresses.

[#8]

I'm terribly sorry for the necro-bump. The issue I'm experiencing seems to be the same one the OP was facing.

We're running CDH 6.3, and I encountered a problem when trying to install the Kudu Python client. I created the symlinks as recommended by mpercy, but I'm still unable to install the kudu-python client.

Our cluster does not have direct access to the internet, so when possible we use an offline install. The official documentation says that the Kudu C++ client libraries and headers are needed for the Kudu Python client. On Oracle Linux 7 trying to install devtoolset-3-toolchain ends in a failure, as a number of dependencies are missing:

Spoiler

Error: Package: devtoolset-3-gdb-7.8.2-38.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libpython2.6.so.1.0()(64bit)
Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libmpfr.so.1()(64bit)
Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libgmp.so.3()(64bit)
Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libgmp.so.3()(64bit)
Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libmpfr.so.1()(64bit)
Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libgmp.so.3()(64bit)
Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libmpfr.so.1()(64bit)

Disregarding that, running pip install --no-index --find-links file:///data0/home/jkovacs/kudu-python-1.10.0.tar.gz  results in the following errors:

Spoiler

   ERROR: Command errored out with exit status 1:
     command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rK2Qbk/kudu-python/pip-egg-info
         cwd: /tmp/pip-install-rK2Qbk/kudu-python/
    Complete output (43 lines):
    Building from system prefix /usr/local
    /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/client.pxd
      tree = Parsing.p_module(s, pxd, full_module_name)
    /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/schema.pxd
      tree = Parsing.p_module(s, pxd, full_module_name)
    /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/errors.pxd
      tree = Parsing.p_module(s, pxd, full_module_name)
    Compiling kudu/client.pyx because it depends on kudu/config.pxi.
    Compiling kudu/errors.pyx because it depends on /usr/lib64/python2.7/site-packages/Cython/Includes/libcpp/string.pxd.
    Compiling kudu/schema.pyx because it depends on kudu/config.pxi.
    [1/3] Cythonizing kudu/client.pyx
    [2/3] Cythonizing kudu/schema.pyx
    [3/3] Cythonizing kudu/errors.pyx
    WARNING: The wheel package is not available.
    DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at <a href="<a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a>" target="_blank"><a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a</a>>
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3910>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3110>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3090>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e50>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e90>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    ERROR: Could not find a version that satisfies the requirement pytest-runner (from versions: none)
    ERROR: No matching distribution found for pytest-runner
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-rK2Qbk/kudu-python/setup.py", line 216, in <module>
        test_suite="kudu.tests"
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/usr/lib/python2.7/site-packages/setuptools/installer.py", line 121, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['/usr/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp8eqkO2', '--quiet', 'pytest-runner']' returned non-zero exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I'd like to ask for some guidance, what to try next?

Thank you!