Read data from Kudu via Spark

Tomas79 — Fri, 16 Sep 2022 12:47:59 GMT

Hi,

I did not found any documents describing how the spark tasks are assigned to which exectur when data is read from Kudu into dataframes. I noticed, that in some cases (did not have enough time to test) Spark reads data ONLY from the Leaders of the tablets, so moving data across network.

Is there any setting or configuration for co-locate the spark task in an executor with a Kudu tablet?

Based on the Kudu documentation, the LEADER is for write, but the FOLLOWERs can server reads too..

Thanks

Re: Read data from Kudu via Spark

awong — Mon, 29 Jan 2018 19:15:53 GMT

The Spark bindings for Kudu 1.5 and below would only scan the leader. Starting in Kudu 1.6, the Spark bindings will have an option to specify scans from the closest replica. This was filed as KUDU-1454 and the change was merged as commit 9f26c9d15 and commit 3abca98c5

question Read data from Kudu via Spark in Archives of Support Questions (Read Only)

Read data from Kudu via Spark

Re: Read data from Kudu via Spark