Reply
Highlighted
Explorer
Posts: 7
Registered: ‎08-08-2017

Caused by: com.cloudera.com.amazonaws.AmazonClientException: Unable to verify integrity of data uplo

[ Edited ]

I am sending data from cloudera flume to s3 with KMS encryption enabled on the s3 bucket. I am getting the following error in the logs .Can you let me know how to fix this error

 

ERROR

org.apache.flume.sink.hdfs.HDFSEventSink

process failed

com.cloudera.com.amazonaws.AmazonClientException: Unable to verify integrity of data upload.  Client calculated content hash (contentMD5: wUgHwMRtgfaGjWED3dN6Yg== in base 64) didn't match hash (etag: 7a59eb560f87364bcc053b3053205f71 in hex) calculated by Amazon S3.  You may need to delete the data stored in Amazon S3. (metadata.contentMD5: wUgHwMRtgfaGjWED3dN6Yg==, md5DigestStream: null, bucketName: encryption-test, key: test/test1.avro

        at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1492)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Nov 18, 10:35:38.278 AM

ERROR

org.apache.flume.SinkRunner

Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: com.cloudera.com.amazonaws.AmazonClientException: Unable to verify integrity of data upload.  Client calculated content hash (contentMD5: wUgHwMRtgfaGjWED3dN6Yg== in base 64) didn't match hash (etag: 7a59eb560f87364bcc053b3053205f71 in hex) calculated by Amazon S3.  You may need to delete the data stored in Amazon S3. (metadata.contentMD5: wUgHwMRtgfaGjWED3dN6Yg==, md5DigestStream: null, bucketName: encryption-test, key: test/test1.avro)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:745)

Caused by: com.cloudera.com.amazonaws.AmazonClientException: Unable to verify integrity of data upload.  Client calculated content hash (contentMD5: wUgHwMRtgfaGjWED3dN6Yg== in base 64) didn't match hash (etag: 7a59eb560f87364bcc053b3053205f71 in hex) calculated by Amazon S3.  You may need to delete the data stored in Amazon S3. (metadata.contentMD5: wUgHwMRtgfaGjWED3dN6Yg==, md5DigestStream: null, bucketName: encryption-test, key: test/test1.avro)

        at com.cloudera.com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1492)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:131)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:123)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:139)

        at com.cloudera.com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:47)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        ... 1 more

Nov 18, 10:35:43.278 AM

INFO

org.apache.flume.sink.hdfs.BucketWriter

Closing s3a://keys@encryption-test/test/test1.avro

Nov 18, 10:35:43.295 AM

INFO

org.apache.flume.sink.hdfs.BucketWriter

Renaming s3a://keys@encryption-test/test/test1.avro.tmp to s3a://keys@encryption-test/test/test1.avro

Nov 18, 10:35:43.501 AM

WARN

com.cloudera.com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy

Attempting to re-send the request to encryption-test.s3.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.

Announcements