question Convert to parquet format in Archives of Support Questions (Read Only)

Convert to parquet format

gobi_subramani — Fri, 16 Sep 2022 10:47:00 GMT

Hi All,

Using sqoop I am importing data to HDFS on daily basis. Now to increase the perfomance I am gonna use parquet file format.

So, my requirement is, import data from RDBMS and store it in HDFS as parquet file format. wanna know how to convert and is there any best practice to do ?

Re: Convert to parquet format

gkeys — Tue, 08 Nov 2016 21:23:14 GMT

HDP 2.3+ packages Sqoop 1.4.6 which allows direct import to HDFS as parquet file, by using:

--as-parquetfile

See: https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html

If you import directly to hive table (vs HDFS) you may need to do this as 2-step process (https://community.hortonworks.com/questions/56847/parquet-files-sqoop-import.html)