12-04-2017 07:20 PM - last edited on 12-05-2017 05:39 AM by cjervis
My goal is to use oozie, to start a shell action and retrieve an unknown FILENAME of a parquet file in a specific directory.
Ultimately, I will like to create a impala/hive table off of the newly discovered file via an impala-shell all with a few parameters passed to it.
My error lies with retrieving that file name via oozie or a shell action.
ex. Oozie -> Shell Action -> retrieves the (unknown parquet) file name from directory -> passes filename into the .SQL file that is run in impala-shell (the passing of other parameters is already working.
Has anyone successfully done this? I have the oozie workflow sqooping data from SQL into parquet, not knowing what the file name is for the rest of the oozie workflow steps is what is giving me a road block at this point.
Any help would be greatly appreciated!