Read data from Kudu via Spark

[#1]

Hi,

 I did not found any documents describing how the spark tasks are assigned to which exectur when data is read from Kudu into dataframes. I noticed, that in some cases (did not have enough time to test) Spark reads data ONLY from the Leaders of the tablets, so moving data across network.

Is there any setting or configuration for co-locate the spark task in an executor with a Kudu tablet?

Based on the Kudu documentation, the LEADER is for write, but the FOLLOWERs can server reads too..

Thanks

[#2 · Accepted solution]

The Spark bindings for Kudu 1.5 and below would only scan the leader. Starting in Kudu 1.6, the Spark bindings will have an option to specify scans from the closest replica. This was filed as KUDU-1454 and the change was merged as commit 9f26c9d15 and commit 3abca98c5