question Issue with S3 in Support Questions

Issue with S3

Jame1979 — Mon, 10 Apr 2023 15:03:11 GMT

Hello all,
I'm setting up a process in NIFI to download some files from S3 bucket. The process contain a ListS3, UpdateAttribute, FetchS3Object and PutHDFS on a 3 nodes cluster environment. The ListS3 retrieve the files but the files get stuck in the queue before the FetchS3Object and will not move past the FetchS3Object. I don't received any error message and the files would just sit there. Any reason why the files are not moving to the FetchS3Object processor? I have all default settings set on both ListS3 and FetchS3Object.

Thanks!

Re: Issue with S3

VidyaSargur — Tue, 11 Apr 2023 07:43:42 GMT

@Jame1979, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @MattWho @SAMSAL @steven-matison @DigitalPlumber @cotopaul who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Issue with S3

cotopaul — Tue, 11 Apr 2023 08:17:58 GMT

@Jame1979,

It would really help if you could post a screenshot of each of the processors, as the problem might be related to some of your configurations. You might be doing something with UpdateAttribute which will affect the behavior of FetchS3.

As I can see, you have a terminated thread and a running thread, meaning that something happened there. I had a similar issue and I solved it by restarting the NiFi Cluster. There was a problem in the back-end which translated into threads being generated but actually not performing any action. After the restart, everything went back to normal and I started collecting data using Fetch :).

You could also set the FetchS3 to DEBUG to see if any logs are being generated, which might point you into the right direction.

Re: Issue with S3

DigitalPlumber — Tue, 11 Apr 2023 13:56:06 GMT

@Jame1979 , like @cotopaul mentions you seem to have a hung thread that you tried to terminate.
Restart NiFi should solve that.
Additionally and important here is that you are never really "spreading" the work among you 3 nodes.
The List happens on the primary node and it stays on the primary node.
What you should do is right click on the connection between List and UpdateAttribute and select load balancing strategy of "Round Robin"

Re: Issue with S3

jame1997 — Tue, 11 Apr 2023 14:26:00 GMT

Hello Vidya,
Thanks for tagging NIFI experts.

Re: Issue with S3

jame1997 — Tue, 11 Apr 2023 14:28:56 GMT

Hello CotoPaul,
I didn't think to change the FetchS3 to DEBUG, after doing that I received some alerts that stating issue with the VPCE endpoint "unable to get a match with the vpce endpoint pattern; using the configured region: us-east-1". I'm checking with cloudops team on the error message.

Thanks!

Re: Issue with S3

jame1997 — Tue, 11 Apr 2023 14:29:49 GMT

Hello DigitalPlumber,
Thanks for that information will make that changed.

Re: Issue with S3

jame1997 — Thu, 27 Apr 2023 14:51:14 GMT

Hello @MattWho @SAMSAL @steven-matison @DigitalPlumber @cotopaul , after confirming the access I'm able to verified I can access the bucket from aws cli on the same system Nifi is running on and download files.

The problem I'm having now is that the ListS3 pickup the list of the files from the bucket but the FetchS3 doesn't do anything. When I enabled debugging on the FetchS3 I receiving the following error message.

FetchS3 Error message

FetchS3 Configuration

ListS3 Configuration

Any suggestion to what is causing the issue?