livy sql interpreter can not find source file

[#1]

I'm trying to run a SQL query using livy interpreter as below.

%livy.sql
LOAD DATA LOCAL INPATH '/tmp/abc.csv' INTO TABLE abc

it gives this error

org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''/tmp/abc.csv'': No files matching path file:/tmp/abc.csv

But when I run the same thing with %sql as below, it works fine.

%sql
LOAD DATA LOCAL INPATH '/tmp/abc.csv' INTO TABLE abc

Any pointers on this behavior ?

[#2 · Accepted solution]

@yvora Its because %livy.sql interpreter runs in the yarn-cluster mode whereas %sql interpreter runs in the yarn-client mode. Hence %sql can find the local file on the client-machine whereas %livy.sql wont be able to find.

Try putting file in HDFS and use LOAD DATA INPATH rather than LOAD DATA LOCAL INPATH. It should work