Member since
03-14-2023
3
Posts
0
Kudos Received
0
Solutions
03-16-2023
06:38 AM
2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for (auth:KERBEROS)
2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
2023-03-16 16:56:03,064 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.MkTempDir is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:03,436 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.MkTempDir is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:05,049 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.MkTempDir is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:06,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.MkTempDir is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:07,339 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Mkdirs is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:07,638 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Mkdirs is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:07,976 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Mkdirs is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,044 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Mkdirs is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:08,418 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805433_103102715, replicas=192.168.8.4:1004, 192.168.8.1:1004, 192.168.8.5:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:08,446 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Copy is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Copy is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Copy is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:23,479 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805442_103102724, replicas=192.168.8.1:1004, 192.168.8.3:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:38,545 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805447_103102729, replicas=192.168.8.3:1004, 192.168.8.6:1004, 192.168.8.2:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:53,608 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805456_103102738, replicas=192.168.8.2:1004, 192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:08,677 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805462_103102744, replicas=192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:23,742 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805467_103102749, replicas=192.168.8.6:1004 for /data/test/datasync/test.txt
java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.
2023-03-16 16:57:39,289 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Copy is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:39,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Dirmeta is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:40,593 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Dirmeta is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:40,605 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Dirmeta is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:40,651 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Dirmeta is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,347 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node0.complete is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:41,349 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node2.complete is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:41,392 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node1.complete is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,504 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node3.complete is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
... View more
03-16-2023
06:10 AM
@blizano thank your reply. I checked and there is no any firewall rule or network port block. do u have any idea? 2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for (auth:KERBEROS)
2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
2023-03-16 16:56:03,064 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.MkTempDir is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:03,436 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.MkTempDir is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:05,049 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.MkTempDir is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:06,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.MkTempDir is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:07,339 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Mkdirs is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:07,638 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Mkdirs is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:07,976 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Mkdirs is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,044 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Mkdirs is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:08,418 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805433_103102715, replicas=192.168.8.4:1004, 192.168.8.1:1004, 192.168.8.5:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:08,446 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Copy is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Copy is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Copy is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:23,479 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805442_103102724, replicas=192.168.8.1:1004, 192.168.8.3:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:38,545 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805447_103102729, replicas=192.168.8.3:1004, 192.168.8.6:1004, 192.168.8.2:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:53,608 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805456_103102738, replicas=192.168.8.2:1004, 192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:08,677 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805462_103102744, replicas=192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:23,742 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805467_103102749, replicas=192.168.8.6:1004 for /data/test/datasync/test.txt
java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.
2023-03-16 16:57:39,289 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Copy is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:39,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Dirmeta is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:40,593 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Dirmeta is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:40,605 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Dirmeta is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:40,651 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Dirmeta is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,347 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node0.complete is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:41,349 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node2.complete is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:41,392 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node1.complete is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,504 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node3.complete is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
... View more
03-14-2023
06:49 AM
hi
i have issue when i tryin to migrate simple file from s3 to HDFS (onprimes) with AWS DataSync agent.
aws side error:
Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs
cloudwatch:
[ERROR] Deferred error: s3:c96 close("/data/test/datasync//test.txt"): 5 (Input/output error)
\
HDFS name node log:
java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.
... View more
Labels:
- Labels:
-
Cloudera Data Platform (CDP)
-
HDFS