Skip to main content
ganzuoni
Gold ⭐️
May 22, 2026
Solved

Hive export

I'm exporting several tables and I observe that some files are copied to the target path using DistCp (slow) while other files with some other (fast) mean

There is no evidence of the rational behind the choice but the other odd thing is that even if a table is made up of multiple files, hive starts 1 DistCp for each file instead of passing the entire directory

Is there any option to drive the behaviour?

ganzuoni
ganzuoniAnswer
Gold ⭐️
May 22, 2026

Self resolved

The switch between direct single threaded copy and distcp depends on file size gt  hive.exec.copyfile.maxsize

The default value is 32MB

 

 

1 reply

ganzuoni
ganzuoniAuthorAnswer
Gold ⭐️
May 22, 2026

Self resolved

The switch between direct single threaded copy and distcp depends on file size gt  hive.exec.copyfile.maxsize

The default value is 32MB

 

 

Translate