Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Using kudu with Python

Highlighted

Re: Using kudu with Python

Explorer

Thank you very much Mike for the follow-up!

 

Our client wanted us to stop using python, so I cannot verify your solution.

 

The system is CentOS 7.4.1708. 

 

I will return to this Jira issue in future if there is a need to continue with python-kudu.

 

Best Regards

Highlighted

Re: Using kudu with Python

Expert Contributor

With Centos 7 I think the only thing required to make "pip install kudu-python" work out-of-the-box is to add the symlinks I mentioned. I'm going to work on getting those added by default as part of the parcel install in a future release.

Highlighted

Re: Using kudu with Python

Explorer
Nope, that "pip install kudu-python" certainly didn't work for me, claiming
"Cannot find installed kudu client.".
Oviously, kudu was accessible from CLI, so I guess it searches for some
directories that don't get created in parcel install.

Thanks
Highlighted

Re: Using kudu with Python

Expert Contributor

Yes -- I saw the same behavior, and the workaround is adding the symlinks I detailed in a previous comment in this thread.

Highlighted

Re: Using kudu with Python

Explorer

Is there a chance a connect to multiple masters to allow Python client choose the needed leader itself?

 

Current connection string

 

# Connect to Kudu master server
client = kudu.connect(host='kudu.master', port=7051)

 

may prevent some automation tasks, because I have to choose the right leader first.

Highlighted

Re: Using kudu with Python

Cloudera Employee
You can connect to multiple masters by providing a list of master addresses.

Re: Using kudu with Python

Explorer

I'm terribly sorry for the necro-bump. The issue I'm experiencing seems to be the same one the OP was facing.

We're running CDH 6.3, and I encountered a problem when trying to install the Kudu Python client. I created the symlinks as recommended by mpercy, but I'm still unable to install the kudu-python client.

Our cluster does not have direct access to the internet, so when possible we use an offline install. The official documentation says that the Kudu C++ client libraries and headers are needed for the Kudu Python client. On Oracle Linux 7 trying to install devtoolset-3-toolchain ends in a failure, as a number of dependencies are missing:

 

Spoiler
Error: Package: devtoolset-3-gdb-7.8.2-38.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libpython2.6.so.1.0()(64bit)
Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libmpfr.so.1()(64bit)
Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libgmp.so.3()(64bit)
Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libgmp.so.3()(64bit)
Error: Package: devtoolset-3-gcc-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libmpfr.so.1()(64bit)
Error: Package: devtoolset-3-gcc-gfortran-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libgmp.so.3()(64bit)
Error: Package: devtoolset-3-gcc-c++-4.9.2-6.el6.x86_64 (rhscl-devtoolset-3-epel-6-x86_64)
Requires: libmpfr.so.1()(64bit)

Disregarding that, running pip install --no-index --find-links file:///data0/home/jkovacs/kudu-python-1.10.0.tar.gz  results in the following errors:

Spoiler
   ERROR: Command errored out with exit status 1:
     command: /usr/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rK2Qbk/kudu-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rK2Qbk/kudu-python/pip-egg-info
         cwd: /tmp/pip-install-rK2Qbk/kudu-python/
    Complete output (43 lines):
    Building from system prefix /usr/local
    /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/client.pxd
      tree = Parsing.p_module(s, pxd, full_module_name)
    /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/schema.pxd
      tree = Parsing.p_module(s, pxd, full_module_name)
    /usr/lib64/python2.7/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-rK2Qbk/kudu-python/kudu/errors.pxd
      tree = Parsing.p_module(s, pxd, full_module_name)
    Compiling kudu/client.pyx because it depends on kudu/config.pxi.
    Compiling kudu/errors.pyx because it depends on /usr/lib64/python2.7/site-packages/Cython/Includes/libcpp/string.pxd.
    Compiling kudu/schema.pyx because it depends on kudu/config.pxi.
    [1/3] Cythonizing kudu/client.pyx
    [2/3] Cythonizing kudu/schema.pyx
    [3/3] Cythonizing kudu/errors.pyx
    WARNING: The wheel package is not available.
    DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at <a href="<a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a>" target="_blank"><a href="https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a" target="_blank">https://pip.pypa.io/en/latest/development/release-process/#python-2-support</a</a>>
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3910>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3110>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca3090>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e50>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fe8bbca9e90>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/pytest-runner/
    ERROR: Could not find a version that satisfies the requirement pytest-runner (from versions: none)
    ERROR: No matching distribution found for pytest-runner
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-rK2Qbk/kudu-python/setup.py", line 216, in <module>
        test_suite="kudu.tests"
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/usr/lib/python2.7/site-packages/setuptools/installer.py", line 121, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['/usr/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp8eqkO2', '--quiet', 'pytest-runner']' returned non-zero exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I'd like to ask for some guidance, what to try next?

 

 

Thank you!

 

Don't have an account?
Coming from Hortonworks? Activate your account here