PutS3Object fails to upload file to S3 compatible storage with object lock setting

tono425 — Fri, 07 Feb 2025 03:58:20 GMT

Hello,

I'm trying to upload file to S3 compatible IBM Cloud Object Storage by PutS3Object processor.
It fails with the attached error only when target ICOS bucket is configured to protect object.
When we upload file with s3cmd, it works.
Is there any solution for this problem?

Thanks,

2025-02-07 10:17:00,892 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=b4359d36-0193-1000-4303-1924a76d8f5a] Failed to put StandardFlowFileRecord[uuid=3f891ca3-d3b9-4ffa-b446-306386f637db,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1738891020007-580, container=default, section=580], offset=0, length=894211],offset=0,name=89e42cbe-172f-4882-8b9e-695b27c7a6e5,size=894211] to Amazon S3 due to com.amazonaws.services.s3.model.AmazonS3Exception: Missing required content hash for this request: Content-MD5 or x-amz-content-sha256 (Service: Amazon S3; Status Code: 400; Error Code: MissingDigest; Request ID: db2b96e4-d540-4ca9-9dd1-0605ad4b8658; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null com.amazonaws.services.s3.model.AmazonS3Exception: Missing required content hash for this request: Content-MD5 or x-amz-content-sha256 (Service: Amazon S3; Status Code: 400; Error Code: MissingDigest; Request ID: db2b96e4-d540-4ca9-9dd1-0605ad4b8658; S3 Extended Request ID: null; Proxy: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1912) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1450) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1419) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1183) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:838) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:805) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:779) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:735) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:581) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5590) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5537) at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:428) at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6671) at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1913) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1873) at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:615) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)

Re: PutS3Object fails to upload file to S3 compatible storage with object lock setting

tono425 — Fri, 07 Feb 2025 07:00:52 GMT

Additional information.

It looks file upload fails when retention policy is enabled on the target ICOS bucket.

Thanks,

question PutS3Object fails to upload file to S3 compatible storage with object lock setting in Support Questions

PutS3Object fails to upload file to S3 compatible storage with object lock setting

Re: PutS3Object fails to upload file to S3 compatible storage with object lock setting