PySpark - Kudu API/command reference

[#1]

Hi

I have been searching for sometime for a command reference/API manual for PySpark-Kudu and I have been unsuccessful so far. Does Cloudera have something that can be of help?

Thanks.

[#2]

I'm going to take a poke at this and hope I'm not wasting your time...

This looks like a good start:

https://kudu.apache.org/docs/developing.html

I have done minimal Python development using PySpark and Kudu.  It's not too bad...

[#3]

I'm going to take a poke at this and hope I'm not wasting your time...

This looks like a good start:

https://kudu.apache.org/docs/developing.html

I have done minimal Python development using PySpark and Kudu.  It's not too bad...

[#4]

Could you please share how KuduContext is created in PySpark?

I am aware of KUDU-1603, but looking for workarounds and the weird java wrapper detailed in KUDU-1603 is not working as intended. 

[#5 · Accepted solution]

Hi,

I'm not sure there is a full-fledged documentation on Kudu PySpark API: the connector is still in early development phase, if I'm not mistaken.

However, the following in-flight patch has a few examples that might be helpful:

  https://gerrit.cloudera.org/#/c/13102/2/docs/developing.adoc

But it doesn't answer your question about KuduContext: I'm not sure that functionality is implemented at this point.  There was a WIP patch posted some time ago: https://gerrit.cloudera.org/#/c/13086/  However, I don't know how what that status of that work at this point, unfortunately.

[#6]

Whoops, the correct link to the WIP patch for PySpark integration work is

  http://gerrit.cloudera.org:8080/13088

[#7]

Yes - that WIP links back to KUDU-1603 that I shared earlier. Guess, we will have to wait it out. Thanks for your response.