question Re: PySpark - Kudu API/command reference in Support Questions

PySpark - Kudu API/command reference

PrasannaSK — Thu, 15 Aug 2019 19:43:48 GMT

I have been searching for sometime for a command reference/API manual for PySpark-Kudu and I have been unsuccessful so far. Does Cloudera have something that can be of help?

Thanks.

Re: PySpark - Kudu API/command reference

pollard — Thu, 15 Aug 2019 19:11:40 GMT

I'm going to take a poke at this and hope I'm not wasting your time...

This looks like a good start:

https://kudu.apache.org/docs/developing.html

I have done minimal Python development using PySpark and Kudu. It's not too bad...

Re: PySpark - Kudu API/command reference

pollard — Thu, 15 Aug 2019 19:13:15 GMT

I'm going to take a poke at this and hope I'm not wasting your time...

This looks like a good start:

https://kudu.apache.org/docs/developing.html

I have done minimal Python development using PySpark and Kudu. It's not too bad...

Re: PySpark - Kudu API/command reference

PrasannaSK — Thu, 15 Aug 2019 21:12:55 GMT

Could you please share how KuduContext is created in PySpark?

I am aware of KUDU-1603, but looking for workarounds and the weird java wrapper detailed in KUDU-1603 is not working as intended.

Re: PySpark - Kudu API/command reference

Alexey1c — Thu, 22 Aug 2019 20:18:38 GMT

Hi,

I'm not sure there is a full-fledged documentation on Kudu PySpark API: the connector is still in early development phase, if I'm not mistaken.

However, the following in-flight patch has a few examples that might be helpful:

https://gerrit.cloudera.org/#/c/13102/2/docs/developing.adoc

But it doesn't answer your question about KuduContext: I'm not sure that functionality is implemented at this point. There was a WIP patch posted some time ago: https://gerrit.cloudera.org/#/c/13086/ However, I don't know how what that status of that work at this point, unfortunately.

Re: PySpark - Kudu API/command reference

Alexey1c — Thu, 22 Aug 2019 20:20:38 GMT

Whoops, the correct link to the WIP patch for PySpark integration work is

http://gerrit.cloudera.org:8080/13088

Re: PySpark - Kudu API/command reference

PrasannaSK — Thu, 22 Aug 2019 20:50:43 GMT

Yes - that WIP links back to KUDU-1603 that I shared earlier. Guess, we will have to wait it out. Thanks for your response.