If there is a large amount of skew you are going to find a new error... so be careful with this. You may wish to revert back to the small file problem. I should mention that if you want you could use a hive statement to 'rebuild' the table which would invoke your settings that you mentioned in the first post. That isn't ideal, as it's more efficient to just write it correctly the first time. But if you want some magic, and aren't going to do this as an ETL process repeatedly but a one time import that might be an acceptable answer.
... View more