Cloudera Community
tono425
user rank
Contributor

Re: PutS3Object fails to upload file to S3 compati...

by Contributor in Support Questions
‎02-06-2025 11:00 PM
‎02-06-2025 11:00 PM
Additional information. It looks file upload fails when retention policy is enabled on the target ICOS bucket. Thanks, ... View more

PutS3Object fails to upload file to S3 compatible ...

by Contributor in Support Questions
‎02-06-2025 07:58 PM
‎02-06-2025 07:58 PM
Hello, I'm trying to upload file to S3 compatible IBM Cloud Object Storage by PutS3Object processor. It fails with the attached error only when target ICOS bucket is configured to protect object. When we upload file with s3cmd, it works. Is there any solution for this problem? Thanks,   2025-02-07 10:17:00,892 ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.aws.s3.PutS3Object PutS3Object[id=b4359d36-0193-1000-4303-1924a76d8f5a] Failed to put StandardFlowFileRecord[uuid=3f891ca3-d3b9-4ffa-b446-306386f637db,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1738891020007-580, container=default, section=580], offset=0, length=894211],offset=0,name=89e42cbe-172f-4882-8b9e-695b27c7a6e5,size=894211] to Amazon S3 due to com.amazonaws.services.s3.model.AmazonS3Exception: Missing required content hash for this request: Content-MD5 or x-amz-content-sha256 (Service: Amazon S3; Status Code: 400; Error Code: MissingDigest; Request ID: db2b96e4-d540-4ca9-9dd1-0605ad4b8658; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null com.amazonaws.services.s3.model.AmazonS3Exception: Missing required content hash for this request: Content-MD5 or x-amz-content-sha256 (Service: Amazon S3; Status Code: 400; Error Code: MissingDigest; Request ID: db2b96e4-d540-4ca9-9dd1-0605ad4b8658; S3 Extended Request ID: null; Proxy: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1912) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1450) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1419) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1183) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:838) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:805) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:779) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:735) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:581) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:559) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5590) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5537) at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:428) at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6671) at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1913) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1873) at org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:615) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:244) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)     ... View more
Labels:

Re: ConvertRecord fails for some files

by Contributor in Support Questions
‎01-20-2025 12:20 AM
‎01-20-2025 12:20 AM
@cloude  Thank you for your answer. Now I understand that is expected behavior. I'll consider solution. Thanks, ... View more

Re: MergeRecord generates multiple files

by Contributor in Support Questions
‎01-17-2025 07:37 AM
‎01-17-2025 07:37 AM
@MattWho  Thank you for explanation. Now I understand MergeRecord determines to which file each flowfile to be merged by schema information. I'll consider increasing "Minimum number of records" as you recommended. Thanks, ... View more

Re: MergeRecord generates multiple files

by Contributor in Support Questions
‎01-16-2025 07:25 AM
‎01-16-2025 07:25 AM
@MattWho  Thank you for your advice. I already tried to increase "Minimum number of records" and "Max Bin Age" but it didn't resolve the problem. As a test, I placed second MergeRecords next to the first one, and the second processor generated the same number of Outputs as Inputs. So I'm under the impression that MergeRecord checks input records and determine if they can be merged. When I use MergeContent processor instead of MergeRecord, it generates 1 output file as I expected. So I am wondering where this difference comes from. Thanks, ... View more

MergeRecord generates multiple files

by Contributor in Support Questions
‎01-16-2025 12:42 AM
‎01-16-2025 12:42 AM
Hello, I' trying to merge over 20000 parquet files into 1 file with MergeRecord processor. But it generates 7 files. It doesn't seem to be caused by Bin size, Bin age or number of records limitation. Looking into the generated file, it looks that similar log records are in the same file. So I assume MergeRecord processor looks into source record content and determine the target file to be merged into. Is this expected behavior? I don't set value in Correlation Attribute Name field. Thanks, ... View more
Labels:

ConvertRecord fails for some files

by Contributor in Support Questions
‎01-16-2025 12:06 AM
‎01-16-2025 12:06 AM
Hello all, I'm trying to convert many records from json to parquet with ConvertRecord processor. Most succeed, but convert fails for some files with this error.   org.apache.parquet.schema.InvalidSchemaException: Cannot write a schema with an empty group: optional group pop_pools    I assume this is because some json files contain the following field.   "pop_pools": {}   As source record is log data, we can't modify it. Is there any way to avoid this error and convert the records to parquet format? Thanks, ... View more
Labels:

Re: Getting java warning after staring NiFi

by Contributor in Support Questions
‎12-17-2024 04:48 PM
1 Kudo
‎12-17-2024 04:48 PM
1 Kudo
@Shelton  Thank you for your advice. As I use the latest version of NiFi and it requires Java 21, I added the following line in bootstrap.conf and confirmed the warning messages disappeared. java.arg.EnableNativeAccess=--enable-native-access=ALL-UNNAMED  I appreciate your help. Thank you, ... View more

Re: Getting java warning after staring NiFi

by Contributor in Support Questions
‎12-15-2024 06:15 PM
‎12-15-2024 06:15 PM
@MattWho  Thank you for your advice. I installed Java 21 (21.0.5) but similar warnings still persist. Dec 16 11:09:10 myserver nifi.sh[253936]: WARNING: A restricted method in java.lang.foreign.Linker has been called Dec 16 11:09:10 myserver nifi.sh[253936]: WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module Dec 16 11:09:10 myserver nifi.sh[253936]: WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module  Thank you, ... View more

Getting java warning after staring NiFi

by Contributor in Support Questions
‎12-12-2024 09:57 PM
1 Kudo
‎12-12-2024 09:57 PM
1 Kudo
Hello, Recently I setup NiFi 2.0. Everytime I start NiFi, I get below warning messages in /var/log/messages.   Dec 13 12:07:58 myserver nifi.sh[208489]: WARNING: A restricted method in java.lang.foreign.Linker has been called Dec 13 12:07:58 myserver nifi.sh[208489]: WARNING: java.lang.foreign.Linker::downcallHandle has been called by org.apache.lucene.store.PosixNativeAccess in an unnamed module Dec 13 12:07:58 myserver nifi.sh[208489]: WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module Dec 13 12:07:58 myserver nifi.sh[208489]: WARNING: Restricted methods will be blocked in a future release unless native access is enabled   Is there any way to supress these messages? Here are product versions. Red Hat Enterprise Linux 8.8 NiFi 2.0.0 Java OpenJDK 23.0.1.0.11 Regards, ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎02-26-2025 03:30 AM
Community Statistics
Member Since ‎10-22-2024 05:41 PM
Last Visited ‎02-26-2025 03:30 AM
Posts 15
Kudos received 6