livy sql interpreter can not find source file

yvora — Thu, 09 Feb 2017 09:22:06 GMT

I'm trying to run a SQL query using livy interpreter as below.

%livy.sql
LOAD DATA LOCAL INPATH '/tmp/abc.csv' INTO TABLE abc

it gives this error

org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException Line 1:23 Invalid path ''/tmp/abc.csv'': No files matching path file:/tmp/abc.csv

But when I run the same thing with %sql as below, it works fine.

%sql
LOAD DATA LOCAL INPATH '/tmp/abc.csv' INTO TABLE abc

Any pointers on this behavior ?

Re: livy sql interpreter can not find source file

kbadani — Thu, 09 Feb 2017 09:24:17 GMT

@yvora Its because %livy.sql interpreter runs in the yarn-cluster mode whereas %sql interpreter runs in the yarn-client mode. Hence %sql can find the local file on the client-machine whereas %livy.sql wont be able to find.

Try putting file in HDFS and use LOAD DATA INPATH rather than LOAD DATA LOCAL INPATH. It should work

question livy sql interpreter can not find source file in Archives of Support Questions (Read Only)

livy sql interpreter can not find source file

Re: livy sql interpreter can not find source file