Hi, I am importing from Oracle using Sqoop1 (version 1.4.6, CDH 5.7.4). I cannot use Sqoop to write directly to the destination Hive table, since this table has got a custom type mapping, that is different from the one used by Sqoop. To achieve this, I offload data to a temporary uncompressed parquet table, and then I load the destination table using Hive (beeline). This way I can compress it (snappy) and fix the datatypes on the fly. This works correctly. The problem is that I have some tables where one or two fields contains some special chars that break my table. I know this because, after a lot of debug, I have a working solution using Oracle's replace function to replace newlines, tabs and carriage return with a space (' '). So, for the fields I know to be problematic, I write a query that replaces those chars upon extraction, and it works fine. However, this is clearly the wrong approach, since I can have other fields with dirty chars, and also other dirty chars apart from those 3 I am replacing. Also, I moved from flatfile to binary (parquet) for this very reason: to not have to bother with special chars. Isn't Parquet supposed to not to care about the data contained in the fields ?? What am I missing ? What can I do to fix this ? For now, avro is not an option (SAS is not compatibile with it). Thanks
... View more
