if you have a general location you can only use time based scheduling ( every 15 min for example ) you cannot use retention or do late arrivals etc. All the advanced waiting for data to arrive features in oozie are essentially out. Essentially mimicks the datasets in oozie: https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a5._Dataset ... View more

You need to look into the logs. Most likely yarn logs of the Map Task of your Oozie launcher. This contains the sqoop command execution and any errors you would normally see on the command line. You can get them from resourcemanager ( click on your oozie launcher job and go through to the map task or use yarn application -logs. You can find any issues in the actual data transfer in the kicked off Mapreduce job which is a separate job ... View more

Pretty sure its a community feature and not supported yet even if it should have found its way into the release. Also its very limited right now. Can you really replace your scheduling with it? It still seems to need a bit of time in the oven. ... View more

It depends a bit what you do with it. Each time you send the data over the network a native datatype will be much faster. 999999999 would be 4bytes as integer but 10 bytes as a string ( characters plus length). That holds for shuffles and any operation that needs to copy data. ( Although some functions like a Map might not be impacted since the Strings will not be necessarily copied in Java if you only assign them for example ) It also costs you more RAM in the Executor which can be a considerable factor in Spark when doing aggregations/joins/sorts etc. And finally when you actually need to do computations on these columns you would have to cast them which will be a pretty costly operation that you could save yourself otherwise. So depending on your usecase the performance and RAM difference can vary between not really significant and considerable. ... View more

I was about to say its not possible. I assume both tables have a column with the same name which is one of the reasons database schemas often prefix column names with a letter from the table. Like TPCH: lineitem.l_key and orders.o_key. In this case you would have to suck it up and name all columns in the join using the AS statement. However it looks like Hive has some cool/dirty tricks up its sleeve. Regular Expressions to specify column names. Here is my setup: hive> describe tb1; id int name string hive> describe tb2; id int age int You then have to disable the use of quotes in identifiers because that interferes with the regex. hive> set hive.support.quoted.identifiers=none; And now you can use Java Regex to select all columns from the right table that is NOT the key. So essentially you get all columns but the duplicate. If you have non-join key columns that are duplicate you can exclude them and rename them with the AS statement after: hive> create table tb3 as select tb1.*, tb2.`(id)?+.+` from tb1, tb2 where tb1.id = tb2.id; You can see that I select all columns from the left table and then use the `` quotes to specify a regular expression for the columns from the right side I want. The regex is a mean trick essentially asking for any string unless it starts with id ( it says id once or not at all and something following. This means if it matches the full string it will not be matched because the remainder of the regex needs to match something. You could also do (id|id2)?+.+ or (id*)?+.+. This gives me a result table with all columns from the left table and all columns but the key column from the right table. hive> describe tb3; id int name string age int Neat. You never stop learning something new. Hive is really cool. Edit: Actually made a little article out of the question because these regex would have made my life much easier multiple times before. So giving a bit more attention to it seems to make sense: https://community.hortonworks.com/articles/43510/excluding-duplicate-key-columns-from-hive-using-re.html ... View more