NiFi connect to s3

Question

I'm just starting to learn NiFi. I need to read parquet data from s3 bucket, I don’t understand how to set up lists3 and fetchs3object processors for reading data. full path looks like this: s3://inbox/prod/export/date=2022-01-07/user=100/cro.parquet I'll write data to sql database - I don't have problems with it, but not sure)) I tried to configure the lists3 processor myself and I think is not very good bucket inbox aws_access_key_id aws_secret_access_key region US EAST endpoint override URL http://s3.wi-fi.ru:8080

ckumar · Answer

In order to read parquet data from s3 bucket flow would look like : ListS3 -> FetchS3 -> ConvertRecord with parquet reader .

So you are facing issue with ListS3 ? Can you please provide more details , I mean if you have configured with all the required details and upon start processor is throwing any error ? Processor show invalid and need some additional information? snapshot , error stack trace would help .

You can also open Support case at support

Thank You

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.