question NiFi connect to s3 in Support Questions

NiFi connect to s3

single_malt — Tue, 11 Jan 2022 10:06:41 GMT

I'm just starting to learn NiFi. I need to read parquet data from s3 bucket, I don’t understand how to set up lists3 and fetchs3object processors for reading data. full path looks like this: s3://inbox/prod/export/date=2022-01-07/user=100/cro.parquet I'll write data to sql database - I don't have problems with it, but not sure)) I tried to configure the lists3 processor myself and I think is not very good bucket inbox aws_access_key_id aws_secret_access_key region US EAST endpoint override URL http://s3.wi-fi.ru:8080

Re: NiFi connect to s3

ckumar — Fri, 14 Jan 2022 13:13:40 GMT

In order to read parquet data from s3 bucket flow would look like : ListS3 -> FetchS3 -> ConvertRecord with parquet reader .

So you are facing issue with ListS3 ? Can you please provide more details , I mean if you have configured with all the required details and upon start processor is throwing any error ? Processor show invalid and need some additional information? snapshot , error stack trace would help .

You can also open Support case at support

Thank You

Re: NiFi connect to s3

single_malt — Thu, 03 Feb 2022 09:08:31 GMT

Hey! I built the process of getting data from s3, but now I have another problem, when converting, part of the data is lost and not written to the SQL database, and I can’t understand where my error, there are my settings. If you help me it will be very great.
I use split because without it there is a lot of data and Nifi drops
UpdateAttribute I use because some of the values are taken from the path where the files lie on s3

Re: NiFi connect to s3

single_malt — Mon, 17 Jan 2022 17:15:47 GMT

Can you also tell me how to ask data from a fixed date (period), not everything that there are in the bucket, thanks)