Member since
01-28-2016
38
Posts
14
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1430 | 03-11-2016 04:16 PM |
04-11-2018
07:30 PM
1 Kudo
hive.support.concurrency property enables locking. When a queries is shutdown its locks should be released immediately. When dies abruptly it may leave locks behind. These will be cleaned up by a background process running from a standalone Hive metastore process. This process will consider locks abandoned if they have not heartbeated for (by default) 5 minutes. Metastore logfile should have entries from AcidHouseKeeperService - that is the clean up process.
... View more
02-15-2018
12:24 PM
@Daniel Perry You can try to use expression and cat command, something like this (but it won't work if you have comments in hql file, so remove all comments): beeline
-e “$(hdfs dfs -cat <hdfs_filename>)” if you still want to keep comments you can pipeline the output to sed, keep in mind that comments should start from a new line: beeline -e “$(hdfs dfs -cat <hdfs_filename> | sed -e 's/^--.*//g' )' )”
... View more
09-15-2017
05:19 PM
1 Kudo
@Daniel Perry Im not sure why you are trying this way. But instead of that insert statement which you have mentioned try something like this: insert into table ${database}.process_status_stg_${hive_table} select ('${hive_table}','${current_time}','${taskStatus}','${workflowID}','${errorCode}', (CASE ${taskStatus} WHEN 'COMPLETED' THEN '${hiveconf:deltaCount}' ELSE 'N/A' end as recordsLoadedCount)) from dummytable;
Hope it helps!!
... View more
06-14-2016
09:34 AM
@Daniel Perry Oh one last idea if you use sqoop? If you cannot use the sqoop metastore you might be able to run sqoop in a shell/ssh action grep the output of the sqoop incremental import which will display the last value save it in hdfs and use it next time. Of course a bit ugly but ...
... View more
06-03-2016
10:06 PM
2 Kudos
When Falcon creates Oozie bundle, it will pass the nominal time range and frequency to Oozie. Oozie bundle uses these information to create workflows for each nominal time.
... View more
04-15-2016
03:25 PM
@Daniel Perry The below is the regex to get the date out there: [0-9]+(.*)[0-9]+ Now how to do that? Below is some pseudo code on how I once did something similar. Its a lot of trickery with the different exclamation marks ( notice how the `` is used to execute a command and the "" to denotate a string. You also have to escape characters a lot. So its not finished. You may have to play around with it a bit. But it should work. myvar=+----..........2016 .... ; datevar="`echo "$myvar" | sed 'myregex'`" The second possibility is to do that in oozie. Oozie supports the JSTL expression language functions and you could just use substring since the length of the before and after string always seems to be the same? http://beginnersbook.com/2013/12/jstl-substring-substringafter-substringbefore-functions/
... View more
03-11-2016
04:16 PM
2 Kudos
I've resolved this issue, turned out the problem came from a driver that was being used --driver
oracle.jdbc.OracleDriver Removed the driver, problem solved!
... View more
02-04-2016
10:55 AM
@Chris Nauroth This is good information. Thanks for sharing this.
... View more
01-29-2016
11:38 AM
Thanks, this is very helpful in understanding the partitioning. I've made the changes you have suggested but i'm still not getting any data into my table, despite the staging table being populated with data, do you know why this might be the case?
... View more