Cloudera Community

Support Questions

Find answers, ask questions, and share your expertise
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search

Issue on migration: AWS DataSync to HDFS

avatar
author-rank Vako
New Contributor

Created on ‎03-14-2023 06:49 AM - last edited on ‎03-14-2023 09:16 AM by Community Manager Community Manager

hi

i have issue when i tryin to migrate simple file from s3 to HDFS (onprimes) with AWS DataSync agent.

aws side error:

Transfer and verification completed. Verification detected mismatches. Files with mismatches are listed in Cloud Watch Logs

cloudwatch:

[ERROR] Deferred error: s3:c96 close("/data/test/datasync//test.txt"): 5 (Input/output error)

\

HDFS name node log: 

java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.

2,276 Views
0 Kudos
4 REPLIES 4

avatar
author-rank DianaTorres author-rank
Community Manager

Created ‎03-14-2023 09:20 AM

@Vako Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our HDFS experts @blizano and @pajoshi  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Regards,

Diana Torres,
Community Moderator

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
2,247 Views
0 Kudos

avatar
author-rank blizano author-rank
Contributor

Created ‎03-14-2023 10:32 AM

Hello @Vako

 

You may require further investigation from the AWS support team concerning this: 

 

[ERROR] Deferred error: s3:c96 close("/data/test/datasync//test.txt"): 5 (Input/output error)

 

This error comes from S3 protocol, also this message from NN logs:

 

java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.

 

This means that the client reached the Namenode, so it wrote the file at metadata level in HDFS, but when the application tried to write the data to the different datanodes this fails for some reason, maybe you can check if Datanode's logs tell you something at the time of the issue, but I've always seen this issue caused by some problem/misconfiguration in the application side, also you can validate if there are no firewall rules blocking the datanode port for the server where you are running this AWS DataSync agent.

 

Hope this helps. 

 

2,201 Views
0 Kudos

avatar
author-rank Vako
New Contributor

Created ‎03-16-2023 06:10 AM

@blizano thank your reply.

 

I checked and there is no any firewall rule or network port block.

do u have any idea?

 

2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for  (auth:KERBEROS)
2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for  (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
2023-03-16 16:56:03,064 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.MkTempDir is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:03,436 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.MkTempDir is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:05,049 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.MkTempDir is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:06,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.MkTempDir is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:07,339 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Mkdirs is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:07,638 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Mkdirs is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:07,976 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Mkdirs is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,044 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Mkdirs is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:08,418 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805433_103102715, replicas=192.168.8.4:1004, 192.168.8.1:1004, 192.168.8.5:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:08,446 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Copy is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Copy is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Copy is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:23,479 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805442_103102724, replicas=192.168.8.1:1004, 192.168.8.3:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:38,545 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805447_103102729, replicas=192.168.8.3:1004, 192.168.8.6:1004, 192.168.8.2:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:53,608 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805456_103102738, replicas=192.168.8.2:1004, 192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:08,677 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805462_103102744, replicas=192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:23,742 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805467_103102749, replicas=192.168.8.6:1004 for /data/test/datasync/test.txt
java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.
2023-03-16 16:57:39,289 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Copy is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:39,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Dirmeta is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:40,593 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Dirmeta is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:40,605 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Dirmeta is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:40,651 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Dirmeta is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,347 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node0.complete is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:41,349 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node2.complete is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:41,392 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node1.complete is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,504 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node3.complete is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744

 

2,142 Views
0 Kudos

avatar
author-rank Vako
New Contributor

Created ‎03-16-2023 06:38 AM 

2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for  (auth:KERBEROS)
2023-03-16 16:56:02,552 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for  (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdfs.protocol.ClientProtocol
2023-03-16 16:56:03,064 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.MkTempDir is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:03,436 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.MkTempDir is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:05,049 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.MkTempDir is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:06,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.MkTempDir is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:07,339 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Mkdirs is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:07,638 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Mkdirs is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:07,976 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Mkdirs is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,044 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Mkdirs is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:56:08,418 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805433_103102715, replicas=192.168.8.4:1004, 192.168.8.1:1004, 192.168.8.5:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:08,446 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Copy is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Copy is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:56:08,516 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Copy is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:56:23,479 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805442_103102724, replicas=192.168.8.1:1004, 192.168.8.3:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:38,545 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805447_103102729, replicas=192.168.8.3:1004, 192.168.8.6:1004, 192.168.8.2:1004 for /data/test/datasync/test.txt
2023-03-16 16:56:53,608 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805456_103102738, replicas=192.168.8.2:1004, 192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:08,677 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805462_103102744, replicas=192.168.8.5:1004, 192.168.8.6:1004 for /data/test/datasync/test.txt
2023-03-16 16:57:23,742 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1176805467_103102749, replicas=192.168.8.6:1004 for /data/test/datasync/test.txt
java.io.IOException: File /data/test/datasync/test.txt could only be written to 0 of the 1 minReplication nodes. There are 6 datanode(s) running and 6 node(s) are excluded in this operation.
2023-03-16 16:57:39,289 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Copy is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:39,863 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node3.Dirmeta is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2023-03-16 16:57:40,593 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node0.Dirmeta is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:40,605 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node2.Dirmeta is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:40,651 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/0/node1.Dirmeta is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,347 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node0.complete is closed by libhdfs3_client_random_1823718439_count_148_pid_7924_tid_139642409154304
2023-03-16 16:57:41,349 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node2.complete is closed by libhdfs3_client_random_129521979_count_145_pid_7924_tid_139642140587776
2023-03-16 16:57:41,392 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node1.complete is closed by libhdfs3_client_random_1964951034_count_146_pid_7924_tid_139642325227264
2023-03-16 16:57:41,504 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /data/test/datasync/.aws-datasync/task-036cf039adbe9c036/node3.complete is closed by libhdfs3_client_random_1359402376_count_147_pid_7924_tid_139642283263744
2,142 Views
0 Kudos
Announcements
What's New @ Cloudera
Cloudera Data Engineering 1.23: Access Spark from Your Favor...
What's New @ Cloudera
HBase REST server scaling support is Generally Available
What's New @ Cloudera
New CLI option in the update-database command
What's New @ Cloudera
New Action menu item in the Cloudera Operational Database UI
What's New @ Cloudera
Get the list of supported instance types in the Cloudera Ope...
View More Announcements