question Re: How to run beeline command against file on HDFS? in Archives of Support Questions (Read Only)

How to run beeline command against file on HDFS?

daniel_perry — Fri, 13 Oct 2017 17:29:41 GMT

Hi all,

Sorry for the basic question i've had little success searching online, i just need clarification whether something is possible.

Can i run a beeline command against a file that is in HDFS. I know we use -f in beeline to specify a file when its on the local file system but can this also be done against a file on HDFS.

My use case is that i'd like to run a beeline command through a shell action in Oozie. I'm hitting some issues using Hive2 actions so i wanted to try using a shell action instead

Any help is much appreciated,

Thanks

Re: How to run beeline command against file on HDFS?

YangWang — Fri, 13 Oct 2017 17:56:48 GMT

@Daniel Perry

Beeline expects the HQL file to be local file system. So If your HQL file is in HDFS, you should first download it to local file system, then feed it to Beeline.

Re: How to run beeline command against file on HDFS?

daniel_perry — Fri, 13 Oct 2017 18:13:13 GMT

Thought as much, thanks for confirming

Re: How to run beeline command against file on HDFS?

rodionsvasiljev — Thu, 15 Feb 2018 20:24:43 GMT

@Daniel Perry

You can try to use expression and cat command, something like this (but it won't work if you have comments in hql file, so remove all comments):

beeline -e “$(hdfs dfs -cat <hdfs_filename>)”

if you still want to keep comments you can pipeline the output to sed, keep in mind that comments should start from a new line:

beeline -e “$(hdfs dfs -cat <hdfs_filename> | sed -e 's/^--.*//g' )' )”