question Processing Fixed Width Files in Hive Using Native (Non-UTF8) Character Sets in Archives of Support Questions (Read Only)

Processing Fixed Width Files in Hive Using Native (Non-UTF8) Character Sets

aniruddha_ghosh — Sat, 21 Jul 2018 16:09:21 GMT

Hi,

I have a requirement to load Fixed Width file in hive table where input file is not always UTF-8 encoded.

I found 2 different classes are available for this - 'org.apache.hadoop.hive.serde2.RegexSerDe' to read from fixed width file on defined offset values and 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' for non utf8 encoding. But unable to use them together when creating external table.

Can someone of you please help me with a solution. Thanks in advance!!

Re: Processing Fixed Width Files in Hive Using Native (Non-UTF8) Character Sets

sweeks — Mon, 23 Jul 2018 20:35:17 GMT

I would just read the table with the LazySimpleSerDe and use the substr() function to extract out the columns. I've found that to be more performant than the RegexSerDe and it's clearer to read. You can either run the substring query directly or put it in a view.

Re: Processing Fixed Width Files in Hive Using Native (Non-UTF8) Character Sets

aniruddha_ghosh — Fri, 27 Jul 2018 02:25:18 GMT

Thank you Shawn for your prompt response. I found an alternate way. Did UTF-8 conversion using iconv before reading in external table with RegexSerDe. In my case Hive by default supports UTF-8 charactersets.