Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Read data from Kudu via Spark

Solved Go to solution
Highlighted

Read data from Kudu via Spark

Master Collaborator

Hi,

 I did not found any documents describing how the spark tasks are assigned to which exectur when data is read from Kudu into dataframes. I noticed, that in some cases (did not have enough time to test) Spark reads data ONLY from the Leaders of the tablets, so moving data across network.

 

Is there any setting or configuration for co-locate the spark task in an executor with a Kudu tablet?

Based on the Kudu documentation, the LEADER is for write, but the FOLLOWERs can server reads too..

 

Thanks

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Read data from Kudu via Spark

Contributor

The Spark bindings for Kudu 1.5 and below would only scan the leader. Starting in Kudu 1.6, the Spark bindings will have an option to specify scans from the closest replica. This was filed as KUDU-1454 and the change was merged as commit 9f26c9d15 and commit 3abca98c5

View solution in original post

1 REPLY 1

Re: Read data from Kudu via Spark

Contributor

The Spark bindings for Kudu 1.5 and below would only scan the leader. Starting in Kudu 1.6, the Spark bindings will have an option to specify scans from the closest replica. This was filed as KUDU-1454 and the change was merged as commit 9f26c9d15 and commit 3abca98c5

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here