Skip to main content
PrasannaSK
PrasannaSK
Silver ⭐️⭐️⭐️
August 15, 2019
Solved

PySpark - Kudu API/command reference

Hi

 

I have been searching for sometime for a command reference/API manual for PySpark-Kudu and I have been unsuccessful so far. Does Cloudera have something that can be of help?

 

Thanks.

Alexey1c
Alexey1cAnswer
Gold ⭐️
August 22, 2019

Hi,

 

I'm not sure there is a full-fledged documentation on Kudu PySpark API: the connector is still in early development phase, if I'm not mistaken.

 

However, the following in-flight patch has a few examples that might be helpful:

  https://gerrit.cloudera.org/#/c/13102/2/docs/developing.adoc

 

But it doesn't answer your question about KuduContext: I'm not sure that functionality is implemented at this point.  There was a WIP patch posted some time ago: https://gerrit.cloudera.org/#/c/13086/  However, I don't know how what that status of that work at this point, unfortunately.

View in thread

6 replies

pollard
pollard
Gold ⭐️
August 15, 2019

I'm going to take a poke at this and hope I'm not wasting your time...

 

This looks like a good start:

https://kudu.apache.org/docs/developing.html

 

I have done minimal Python development using PySpark and Kudu.  It's not too bad...

 

Translate
pollard
pollard
Gold ⭐️
August 15, 2019

I'm going to take a poke at this and hope I'm not wasting your time...

 

This looks like a good start:

https://kudu.apache.org/docs/developing.html

 

I have done minimal Python development using PySpark and Kudu.  It's not too bad...

 

Translate
PrasannaSK
PrasannaSKAuthor
Silver ⭐️⭐️⭐️
August 15, 2019

Could you please share how KuduContext is created in PySpark?

 

I am aware of KUDU-1603, but looking for workarounds and the weird java wrapper detailed in KUDU-1603 is not working as intended. 

Translate
Alexey1c
Gold ⭐️
Alexey1cAnswer
Gold ⭐️
August 22, 2019

Hi,

 

I'm not sure there is a full-fledged documentation on Kudu PySpark API: the connector is still in early development phase, if I'm not mistaken.

 

However, the following in-flight patch has a few examples that might be helpful:

  https://gerrit.cloudera.org/#/c/13102/2/docs/developing.adoc

 

But it doesn't answer your question about KuduContext: I'm not sure that functionality is implemented at this point.  There was a WIP patch posted some time ago: https://gerrit.cloudera.org/#/c/13086/  However, I don't know how what that status of that work at this point, unfortunately.

Translate
Terms & ConditionsAccessibility statement