How to run HQL file in Spark

[#1]

I want to read a hql file in spark job. This hql creates a table by joining 3-4 other tables.

I don't want to write the sql statement in the spark job instead I want to pass HQL file as an argument to spark job and then run the hql file.

Is it possible in Spark ?

[#2]

@Amit Kumar Agarwal

See the link below:

https://hadoopist.wordpress.com/2016/03/12/how-to-execute-hive-sql-file-in-spark-engine/

[#3]

@Amit Kumar Agarwal

If you are looking to do it from a program then try something like the below:

http://stackoverflow.com/questions/31313361/sparksql-hql-script-in-file-to-be-loaded-on-python-code

[#4]

This link explains how to execute hive sql using spark-sql shell. But I want to the call the file programatically not through shell.

[#5]

I went through this Stackoverflow link, but I don't see any 'open' API in spark...getting compiler error...

[#6]

"open" is not a spark api command, it is a python command. What language are you using? Replace open("file.hql").read() with the equivalent command/code-block in that language.

[#7]

I'm using scala..couldn't find equivalent "open" API in scala.

[#8 · Accepted solution]

We can use below Scala API to read file:

sqlContext.sql(scala.io.Source.fromFile("/vzhome/agaram8/HQLScripts/count.hql").getLines

[#9]

Hi

Amit Kumar Agarwal

I am looking to run Hive HQL from SPARK SQL.. could you please provide the guidance for same.

Thanks,

Deepesh

[email protected]