Using kudu with Python

[#1]

Hi,

I am unable to use kudu with Python.

[cloudera@quickstart ~]$ sudo pip install kudu-python
Collecting kudu-python
Using cached kudu-python-1.2.0.tar.gz
Complete output from command python setup.py egg_info:
Cannot find installed kudu client.

----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-GE6kA6/kudu-python/

[cloudera@quickstart ~]$ kudu -version
kudu 1.2.0-cdh5.10.2
revision f8f159f3c55f72ef026776f6176855e4f6e70271
build type RELEASE
built by jenkins at 07 Jun 2017 01:01:28 PST on kudu-centos66-19fa.vpc.cloudera.com
build id 2017-06-07_00-28-13

[cloudera@quickstart ~]$ python -V
Python 2.7.13 :: Anaconda 4.4.0 (64-bit)

This is what I am following:

https://www.cloudera.com/documentation/kudu/latest/topics/kudu_development.html

Any help?

[#2]

You need to first install the Kudu client system packages. Check out the documentation here: https://www.cloudera.com/documentation/kudu/latest/topics/kudu_installation.html#install_cmdline

Specifically, you need to install the 'kudu-client0' (RPM distros) or 'libkuduclient0' (DEB distros) package.

[#3]

Hi,

How do I find "kudu.master"?  

client = kudu.connect(host='kudu.master', port=7051)

I get an error when I use it as a localhost:

>>> client = kudu.connect(host='localhost', port=7051)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/cloudera/anaconda2/lib/python2.7/site-packages/kudu/__init__.py", line 92, in connect
rpc_timeout_ms=rpc_timeout_ms)
File "kudu/client.pyx", line 271, in kudu.client.Client.__cinit__ (kudu/client.cpp:5415)
File "kudu/errors.pyx", line 62, in kudu.errors.check_status (kudu/errors.cpp:1073)
kudu.errors.KuduBadStatus: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.0.1:7051, num_attempts: 241) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.0.1:7051: connect: Connection refused (error 111)
 

[#4]

Bear in mind that this Python code merely connects and operates on an existing Kudu cluster; it doesn't start one for you. If you haven't yet created and started a Kudu service, you'll need to do that first.

If you have, the value for 'host' in kudu.connect() needs to be the valid hostname where you're running the Kudu master.

[#5]

Hi Adar,

My CDH cluster verison is 5.11.0.

I follow the steps to install kudu-client0,kudu-client-devel first.

rpm -qa|grep kudu
kudu-client-devel-1.3.0+cdh5.11.0+0-1.cdh5.11.0.p0.13.el6.x86_64
kudu-client0-1.3.0+cdh5.11.0+0-1.cdh5.11.0.p0.13.el6.x86_64

But when i install the kudu-python using the source packge (kudu-python-1.2.0). someother errors show up,

Could you help take a look ? 

Thanks

/opt/cloudera/parcels/Anaconda/bin/python setup.py install
Building from system prefix /usr
running install
running bdist_egg
running egg_info
writing requirements to kudu_python.egg-info/requires.txt
writing kudu_python.egg-info/PKG-INFO
writing top-level names to kudu_python.egg-info/top_level.txt
writing dependency_links to kudu_python.egg-info/dependency_links.txt
reading manifest file 'kudu_python.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '../LICENSE.txt'
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '#*' found anywhere in distribution
warning: no previously-included files matching '.git*' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
writing manifest file 'kudu_python.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying kudu/version.py -> build/lib.linux-x86_64-2.7/kudu
running build_ext
building 'kudu.client' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include -I/opt/cloudera/parcels/Anaconda/include/python2.7 -c kudu/client.cpp -o build/temp.linux-x86_64-2.7/kudu/client.o
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
g++ -pthread -shared -L/tmp/tmpQcQ0Zo/Anaconda-4.0.0/lib -Wl,-rpath=/tmp/tmpQcQ0Zo/Anaconda-4.0.0/lib,--no-as-needed build/temp.linux-x86_64-2.7/kudu/client.o -L/usr/lib -L/tmp/tmpQcQ0Zo/Anaconda-4.0.0/lib -Wl,-R/usr/lib -lkudu_client -lpython2.7 -o build/lib.linux-x86_64-2.7/kudu/client.so
/usr/bin/ld: cannot find -lpython2.7
collect2: ld returned 1 exit status
error: command 'g++' failed with exit status 1

[#6]

Looks like you don't have the Python development packages installed. On my Ubuntu system that's the libpythonx.y-dev package, where 'x' is 2 and 'y' is 7. On Red Hat or Cent OS systems I think you want to install the python-devel package. On my test Cent OS 6.6 system python-devel provides /usr/lib64/libpython2.6.so.

[#7]

Hello Adar,

That link leads to 404.

We have Kudu parcel. What should we do now in order to install the python-kudu via pip?

Thanks

[#8]

Hi elkarel,

This is a very old thread and has some outdated information and expired links.

Please include any problems you are facing including any error messages you are seeing, your version, platform, etc.

Regarding your question about the documentation, there is no longer any special procedure to install Kudu. With the latest versions of CDH, Kudu is included in the parcel: https://www.cloudera.com/documentation/enterprise/5-14-x/topics/kudu_install_cm.html

Thanks,

Mike

[#9]

I looked at this a little more and I think kudu-python install on RHEL 6 / Centos 6 is broken because of the lack of Python 2.6 support in newer versions of pip. There are a couple of related JIRAs on this issue: https://issues.apache.org/jira/browse/KUDU-1705 and https://issues.apache.org/jira/browse/KUDU-2442

Another issue is that the parcel does not link the include files and libraries into a place that pip can find them. You may be able to work around that with putting symlinks from /opt/cloudera/parcels/CDH/includes/kudu into /usr/include/kudu and /opt/cloudera/parcels/CDH/lib64/*kudu* into /usr/lib64/ to see if that helps solve the problem on a newer OS version.

[#10]

I looked into this a little more today, out of interest, and added some additional info about how to make it work on EL6 here: https://issues.apache.org/jira/browse/KUDU-2442

Regardless, if you are using a parcel install, you must first symlink the include files and the libraries into /usr/ or /usr/local, something like this:

sudo ln -s /opt/cloudera/parcels/CDH/include/kudu /usr/local/include/
sudo ln -s /opt/cloudera/parcels/CDH/lib64/libkudu_client.so /usr/local/lib64/
sudo ln -s /opt/cloudera/parcels/CDH/lib64/libkudu_client.so.0 /usr/local/lib64/
sudo ln -s /opt/cloudera/parcels/CDH/lib64/libkudu_client.so.0.1.0 /usr/local/lib64/