question Re: How to run HQL file in Spark in Archives of Support Questions (Read Only)

How to run HQL file in Spark

bigdataam09 — Sat, 29 Oct 2016 03:20:48 GMT

I want to read a hql file in spark job. This hql creates a table by joining 3-4 other tables.

I don't want to write the sql statement in the spark job instead I want to pass HQL file as an argument to spark job and then run the hql file.

Is it possible in Spark ?

Re: How to run HQL file in Spark

egarelnabi — Sat, 29 Oct 2016 04:11:35 GMT

@Amit Kumar Agarwal

See the link below:

https://hadoopist.wordpress.com/2016/03/12/how-to-execute-hive-sql-file-in-spark-engine/

Re: How to run HQL file in Spark

bigdataam09 — Mon, 31 Oct 2016 21:04:49 GMT

This link explains how to execute hive sql using spark-sql shell. But I want to the call the file programatically not through shell.

Re: How to run HQL file in Spark

egarelnabi — Mon, 31 Oct 2016 21:38:28 GMT

@Amit Kumar Agarwal

If you are looking to do it from a program then try something like the below:

http://stackoverflow.com/questions/31313361/sparksql-hql-script-in-file-to-be-loaded-on-python-code

Re: How to run HQL file in Spark

bigdataam09 — Mon, 31 Oct 2016 21:47:30 GMT

I went through this Stackoverflow link, but I don't see any 'open' API in spark...getting compiler error...

Re: How to run HQL file in Spark

egarelnabi — Mon, 31 Oct 2016 22:18:45 GMT

"open" is not a spark api command, it is a python command. What language are you using? Replace open("file.hql").read() with the equivalent command/code-block in that language.

Re: How to run HQL file in Spark

bigdataam09 — Tue, 01 Nov 2016 01:29:38 GMT

I'm using scala..couldn't find equivalent "open" API in scala.

Re: How to run HQL file in Spark

bigdataam09 — Tue, 01 Nov 2016 01:44:13 GMT

We can use below Scala API to read file:

sqlContext.sql(scala.io.Source.fromFile("/vzhome/agaram8/HQLScripts/count.hql").getLines

Re: How to run HQL file in Spark

deepeshnema — Tue, 24 Oct 2017 18:24:26 GMT

Amit Kumar Agarwal

I am looking to run Hive HQL from SPARK SQL.. could you please provide the guidance for same.

Thanks,

Deepesh

deepeshnema@gmail.com