Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Read data from Kudu via Spark

avatar

Hi,

 I did not found any documents describing how the spark tasks are assigned to which exectur when data is read from Kudu into dataframes. I noticed, that in some cases (did not have enough time to test) Spark reads data ONLY from the Leaders of the tablets, so moving data across network.

 

Is there any setting or configuration for co-locate the spark task in an executor with a Kudu tablet?

Based on the Kudu documentation, the LEADER is for write, but the FOLLOWERs can server reads too..

 

Thanks

 

1 ACCEPTED SOLUTION

avatar
Rising Star

The Spark bindings for Kudu 1.5 and below would only scan the leader. Starting in Kudu 1.6, the Spark bindings will have an option to specify scans from the closest replica. This was filed as KUDU-1454 and the change was merged as commit 9f26c9d15 and commit 3abca98c5

View solution in original post

1 REPLY 1

avatar
Rising Star

The Spark bindings for Kudu 1.5 and below would only scan the leader. Starting in Kudu 1.6, the Spark bindings will have an option to specify scans from the closest replica. This was filed as KUDU-1454 and the change was merged as commit 9f26c9d15 and commit 3abca98c5