ERROR [partition-dump-thread-82]: tools.DistCp (DistCp.java:run(133)) - Duplicate files in input path

[#1]

Hello,

I'm trying to export hive table and the export is skipping the buckets in 2019-01-03 partition for 2 tables - 'users_info_af_a' and 'users_sysinfo_af_a'

Command we used to do this:

export table users_sysinfo_af_a to 'hdfs://clustername/user/test/exp/audit.users_sysinfo_af_a';

Hive log captures below error:

2019-07-31 12:44:51,462 INFO [partition-dump-thread-99]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1194)) - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb

2019-07-31 12:44:51,462 INFO [partition-dump-thread-99]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1194)) - io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor

2019-07-31 12:44:51,527 ERROR [partition-dump-thread-99]: tools.DistCp (DistCp.java:run(133)) - Duplicate files in input path:

org.apache.hadoop.tools.CopyListing$DuplicateFileException: File hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000 and hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000 would cause duplicates. Aborting

at org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:165)

at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)

at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)

at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)

The hdfs file exists and i can access as listed below:

hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000

hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000

hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000

hive@hostname:~>

hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000

hive@hostname:~>

hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/

Found 25 items

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000

-rw-r--r-- 3 nifi hadoop 20675229 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00001

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00002

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00003

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00004

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00005

-rw-r--r-- 3 nifi hadoop 9160 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00006

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00007

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00008

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00009

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00010

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00011

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00012

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00013

-rw-r--r-- 3 nifi hadoop 13076 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00014

-rw-r--r-- 3 nifi hadoop 15306 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00015

-rw-r--r-- 3 nifi hadoop 18487 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00016

-rw-r--r-- 3 nifi hadoop 18891 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00017

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00018

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00019

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00020

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00021

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00022

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00023

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00024

hive@hostname:~>

hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/

Found 25 items

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000

-rw-r--r-- 3 nifi hadoop 20615657 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00001

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00002

-rw-r--r-- 3 nifi hadoop 23901 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00003

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00004

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00005

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00006

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00007

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00008

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00009

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00010

-rw-r--r-- 3 nifi hadoop 12900 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00011

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00012

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00013

-rw-r--r-- 3 nifi hadoop 64126 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00014

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00015

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00016

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00017

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00018

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00019

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00020

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00021

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00022

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00023

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00024

hive@hostname:~>

hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/

Found 5 items

-rw-r--r-- 3 nifi hadoop 4 2019-01-05 00:42 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/_orc_acid_version

drwxrwxrwx - nifi hadoop 0 2019-01-05 00:44 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0327737_0327836

drwxrwxrwx - nifi hadoop 0 2019-01-05 00:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0327837_0327936

drwxrwxrwx - nifi hadoop 0 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636

drwxrwxrwx - nifi hadoop 0 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736

hive@hostname:~>