Member since
03-22-2020
2
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
10863 | 03-25-2020 10:38 AM |
03-25-2020
10:38 AM
Below are the steps to troubleshoot distcp:- 1. it is not problem with the hdfs or Kerberos or distcp but a MapReduce. 2. We tried to run a sample MR job to test, then it failed with the following exception Error: Java.io.IOException: initialization of all the collectors failed. Error in last collector was:java.io.IOException: Invalid “mapreduce.task.io.sort.mb”:3276. (The total amount of buffer memory to use while sorting files, in MB). It was expecting less than 2048. Changing this property able to run the distcp smooth. I want to take a moment and say thanks to Shelton for responding it on time.
... View more
03-22-2020
06:44 PM
INFO tools.DistCp: Input Options: DistCpOptions ooxx INFO client.AHSProxy: Connecting to Application History server at ooxx INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 256 for oo at xx INFO security.TokenCache: Got dt for hdfs://clusterA:8020; Kind: HDFS_DELEGATION_TOKEN, Service: ... INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1; dirCnt = 0 INFO tools.SimpleCopyListing: Build file listing
... View more
Labels:
- Labels:
-
Apache Hadoop