question Re: Change default output filename part-r-00000.snappy.parquet in Support Questions

Change default output filename part-r-00000.snappy.parquet

Josh2023 — Fri, 19 May 2023 01:21:58 GMT

Hello,

I am trying to run a mapreduce job and output it on a folder. Is there any way to change the default filename. My default filename is part-r-00000.snappy.parquet and I want to add a date to this. Is this possible.

Is there a parameter to change this, thank you.

Re: Change default output filename part-r-00000.snappy.parquet

VidyaSargur — Fri, 19 May 2023 06:11:09 GMT

@Josh2023, Welcome to our community! To help you get the best possible answer, I have tagged our MapR experts @asish @mugdha who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Change default output filename part-r-00000.snappy.parquet

asish — Fri, 19 May 2023 06:14:52 GMT

@Josh2023 can you try below:

SET hive.exec.compress.output=false; SET hive.exec.dynamic.partition.mode=nonstrict; -- Create a temporary table or select data from existing tables INSERT OVERWRITE DIRECTORY '/path/to/output_directory/filename_' || from_unixtime(unix_timestamp(), 'yyyyMMdd') || '.csv' SELECT * FROM your_table;

Re: Change default output filename part-r-00000.snappy.parquet

asish — Fri, 19 May 2023 07:16:36 GMT

@Josh2023 could you please accept as solution,if this has fixed your issue?

Re: Change default output filename part-r-00000.snappy.parquet

Josh2023 — Fri, 19 May 2023 08:09:33 GMT

Hell @asish, thanks for the answer, unfortunately my output is already parquet format,

Example below is the usually output of my mapreduce job:

part-r-00000.snappy.parquet

I need to use below format as an example:

part-r-00000.2023-05-19-04-09.snappy.parquet

Is this possible?

Re: Change default output filename part-r-00000.snappy.parquet

asish — Fri, 19 May 2023 08:17:57 GMT

this is possible,please try parquet,instead of csv

Re: Change default output filename part-r-00000.snappy.parquet

DianaTorres — Mon, 29 May 2023 12:19:39 GMT

@Josh2023 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

Re: Change default output filename part-r-00000.snappy.parquet

ggangadharan — Tue, 06 Jun 2023 12:13:16 GMT

Since the output file is .parquet , hope you're using ParquetOutputFormat in the MR job config. In that case ParquetOutputFormat.setOutputname method will help to set the base name of the output file.

Ref -
https://www.javadoc.io/doc/org.apache.parquet/parquet-hadoop/1.12.2/org/apache/parquet/hadoop/ParquetOutputFormat.html
https://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.html#setOutputName(org.apache.hadoop.mapreduce.JobContext,%20java.lang.String)

Re: Change default output filename part-r-00000.snappy.parquet

ggangadharan — Tue, 06 Jun 2023 17:14:56 GMT

mapreduce.output.basename also works since as part of setoutput name assigning the same.

Code snippet form ParquetOutputFormat.

protected static void setOutputName(JobContext job, String name) { job.getConfiguration().set("mapreduce.output.basename", name); }

JOB CONF -

Configuration conf = getConf(); conf.set("mapreduce.output.basename","parquet_output");

Output

[hive@c1757-node3 ~]$ hdfs dfs -ls /tmp/parquet-sample Found 4 items -rw-r--r-- 2 hive supergroup 0 2023-06-06 17:08 /tmp/parquet-sample/_SUCCESS -rw-r--r-- 2 hive supergroup 271 2023-06-06 17:08 /tmp/parquet-sample/_common_metadata -rw-r--r-- 2 hive supergroup 1791 2023-06-06 17:08 /tmp/parquet-sample/_metadata -rw-r--r-- 2 hive supergroup 2508 2023-06-06 17:08 /tmp/parquet-sample/parquet_output-m-00000.parquet

Re: Change default output filename part-r-00000.snappy.parquet

DianaTorres — Fri, 09 Jun 2023 18:59:27 GMT

@Josh2023 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.