hive.support.concurrency property enables locking. When a queries is shutdown its locks should be released immediately. When dies abruptly it may leave locks behind. These will be cleaned up by a background process running from a standalone Hive metastore process. This process will consider locks abandoned if they have not heartbeated for (by default) 5 minutes. Metastore logfile should have entries from AcidHouseKeeperService - that is the clean up process. ... View more

@Daniel Perry You can try to use expression and cat command, something like this (but it won't work if you have comments in hql file, so remove all comments): beeline -e “$(hdfs dfs -cat <hdfs_filename>)” if you still want to keep comments you can pipeline the output to sed, keep in mind that comments should start from a new line: beeline -e “$(hdfs dfs -cat <hdfs_filename> | sed -e 's/^--.*//g' )' )” ... View more

@Daniel Perry Im not sure why you are trying this way. But instead of that insert statement which you have mentioned try something like this: insert into table ${database}.process_status_stg_${hive_table} select ('${hive_table}','${current_time}','${taskStatus}','${workflowID}','${errorCode}', (CASE ${taskStatus} WHEN 'COMPLETED' THEN '${hiveconf:deltaCount}' ELSE 'N/A' end as recordsLoadedCount)) from dummytable; Hope it helps!! ... View more

@Daniel Perry Oh one last idea if you use sqoop? If you cannot use the sqoop metastore you might be able to run sqoop in a shell/ssh action grep the output of the sqoop incremental import which will display the last value save it in hdfs and use it next time. Of course a bit ugly but ... ... View more

@Daniel Perry The below is the regex to get the date out there: [0-9]+(.*)[0-9]+ Now how to do that? Below is some pseudo code on how I once did something similar. Its a lot of trickery with the different exclamation marks ( notice how the `` is used to execute a command and the "" to denotate a string. You also have to escape characters a lot. So its not finished. You may have to play around with it a bit. But it should work. myvar=+----..........2016 .... ; datevar="`echo "$myvar" | sed 'myregex'`" The second possibility is to do that in oozie. Oozie supports the JSTL expression language functions and you could just use substring since the length of the before and after string always seems to be the same? http://beginnersbook.com/2013/12/jstl-substring-substringafter-substringbefore-functions/ ... View more

@Chris Nauroth This is good information. Thanks for sharing this. ... View more

Thanks, this is very helpful in understanding the partitioning. I've made the changes you have suggested but i'm still not getting any data into my table, despite the staging table being populated with data, do you know why this might be the case? ... View more