Member since
09-25-2015
7
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2736 | 08-16-2016 06:24 PM |
02-16-2017
07:14 PM
As Eugene suggested, could you paste the output of "dfs -lsr" here so that we can see which dirs are owned by whom? A few other things we need to confirm: Is streaming being used before and after the upgrade? When you say compaction fails, what triggered the compaction? Is that triggered by the system automatically, or is it run by some user manually? If it's a manual compaction, then which user issued the command? You mentioned the problematic table partition was created on 2.5.3. Which user created it? Do you have issue compacting pre-existing tables created on 2.3.6?
... View more
08-16-2016
06:24 PM
2 Kudos
Hive (internal) tables are meant to be fully managed by Hive for both data and metadata (schema). That's not true for external tables. External tables means Hive doesn't own the data per se, but only shares it as one of the applications. Since ACID will need to have complete control of the data, for example, it needs to manage the directory layout, perform compaction, clean up old files and so on, we want to avoid potential interference issues. I understand in your case you may only use the data for Hive only, but the fact that by design external tables can be used by anything else provides enough evidence that disabling ACID on them has to be enforced. As to your use case, you may want to solve the governance issues by creating difference roles or even better, use Ranger to make things easier: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Sys_Admin_Guides/content/ch_hive_auth.html
... View more
04-15-2016
05:58 PM
As far as I understand, tblproperties is just a list of key/value pairs. You can specify whatever you want on it (even if it's not predefined, e.g. 'weather'='warm'). But of course there must be code logic that interprets your tblproperties and do something, otherwise it's useless.
... View more