- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Inserting to text table compressed
- Labels:
-
Apache Impala
Created on 09-19-2017 05:48 PM - edited 09-16-2022 05:16 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I see no way to specify a compression default in the create table statement, so I tried:
SET COMPRESSION_CODEC=gzip;insert overwrite <text_table> select .. from <another_table>
and got "Writing to compressed text table is not supported. Use query option ALLOW_UNSUPPORTED_FORMATS to override." but ALLOW_UNSUPPORTED_FORMATS shouldn't be used according to docs.
Is there a trick to having impala write text files compressed?
-Mauricio
Created 09-20-2017 04:53 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
https://www.cloudera.com/documentation/enterprise/latest/topics/impala_file_formats.html
It mentioned below:
For text format, if LZO compression is used, you must create the table and load data in Hive.
If other kinds of compression are used, you must load data through LOAD DATA, Hive, or manually in HDFS.
So the short answer is that you can't do it in Impala.
Created 09-20-2017 04:53 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
https://www.cloudera.com/documentation/enterprise/latest/topics/impala_file_formats.html
It mentioned below:
For text format, if LZO compression is used, you must create the table and load data in Hive.
If other kinds of compression are used, you must load data through LOAD DATA, Hive, or manually in HDFS.
So the short answer is that you can't do it in Impala.