Support Questions
Find answers, ask questions, and share your expertise

ERROR [partition-dump-thread-82]: tools.DistCp (DistCp.java:run(133)) - Duplicate files in input path

Highlighted

ERROR [partition-dump-thread-82]: tools.DistCp (DistCp.java:run(133)) - Duplicate files in input path

Explorer

Hello,


I'm trying to export hive table and the export is skipping the buckets in 2019-01-03 partition for 2 tables - 'users_info_af_a' and 'users_sysinfo_af_a'

Command we used to do this:

export table users_sysinfo_af_a to 'hdfs://clustername/user/test/exp/audit.users_sysinfo_af_a';

Hive log captures below error:


2019-07-31 12:44:51,462 INFO [partition-dump-thread-99]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1194)) - io.sort.mb is deprecated. Instead, use mapreduce.task.io.sort.mb

2019-07-31 12:44:51,462 INFO [partition-dump-thread-99]: Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1194)) - io.sort.factor is deprecated. Instead, use mapreduce.task.io.sort.factor

2019-07-31 12:44:51,527 ERROR [partition-dump-thread-99]: tools.DistCp (DistCp.java:run(133)) - Duplicate files in input path:

org.apache.hadoop.tools.CopyListing$DuplicateFileException: File hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000 and hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000 would cause duplicates. Aborting

at org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:165)

at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93)

at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)

at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)


The hdfs file exists and i can access as listed below:

hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000

hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000


hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000

-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000


hive@hostname:~>


hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000


hive@hostname:~>


hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/


Found 25 items


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00000


-rw-r--r-- 3 nifi hadoop 20675229 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00001


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00002


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00003


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00004


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00005


-rw-r--r-- 3 nifi hadoop 9160 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00006


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00007


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00008


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00009


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00010


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00011


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00012


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00013


-rw-r--r-- 3 nifi hadoop 13076 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00014


-rw-r--r-- 3 nifi hadoop 15306 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00015


-rw-r--r-- 3 nifi hadoop 18487 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00016


-rw-r--r-- 3 nifi hadoop 18891 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00017


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00018


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00019


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00020


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00021


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00022


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00023


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736/bucket_00024


hive@hostname:~>


hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/


Found 25 items


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00000


-rw-r--r-- 3 nifi hadoop 20615657 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00001


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00002


-rw-r--r-- 3 nifi hadoop 23901 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00003


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00004


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00005


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00006


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00007


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00008


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00009


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00010


-rw-r--r-- 3 nifi hadoop 12900 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00011


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00012


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00013


-rw-r--r-- 3 nifi hadoop 64126 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00014


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00015


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00016


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00017


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00018


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00019


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00020


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00021


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00022


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00023


-rw-r--r-- 3 nifi hadoop 1892 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636/bucket_00024


hive@hostname:~>


hive@hostname:~> hdfs dfs -ls hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/


Found 5 items


-rw-r--r-- 3 nifi hadoop 4 2019-01-05 00:42 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/_orc_acid_version


drwxrwxrwx - nifi hadoop 0 2019-01-05 00:44 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0327737_0327836


drwxrwxrwx - nifi hadoop 0 2019-01-05 00:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0327837_0327936


drwxrwxrwx - nifi hadoop 0 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329537_0329636


drwxrwxrwx - nifi hadoop 0 2019-01-05 01:43 hdfs://clustername/apps/hive/warehouse/audit.db/users_sysinfo_af_a/partition_dt=2019-01-03/delta_0329637_0329736


hive@hostname:~>


Don't have an account?