Member since
04-02-2020
4
Posts
0
Kudos Received
0
Solutions
04-26-2020
09:47 PM
I realize this is a year later, but I thought you might like to know that what you're looking for actually exists. It's not my repo but I've been using them, and plan to update the documentation to make them a little more user friendly. Hope this helps. https://github.com/paulgibeault/SnowflakeNiFiProcessors
... View more
04-22-2020
09:22 AM
Admittedly I know very little about DynamoDB, but I don't think it provides a native method to pull the entire table. If this is a one-time task, I suppose you could export your table to S3, and then use the FetchS3 nifi processor to get it into your flow. If you're looking to process new records as they arrive, I suppose you could use lambda to push the new records into Kafka or Kinesis and use nifi to consume that queue. There is also a way (which I have not explored) to use HiveQL with EMR for querying DynamoDB tables. Assuming that works how I *think* it works, you could then use the SelectHiveQL processor to pull your DynamoDB table into your nifi flow. I've also never used EMR, but maybe it will be a fun experiment?
... View more
04-03-2020
03:08 PM
Thanks for this writeup. I was having a heck of a time authenticating, was expecting the authentication keys to be separate fields in the processor configuration. I didn't realize you could add all that right in the connection string, although it leaves your passwords in plain text. Seems like an opportunity for improvement.
... View more
04-02-2020
08:04 PM
I realize this is from 3 years ago, but nobody ever answered and I needed the answer today. Turns out that if your intent is to push your JSON flow file into your dynamoDB table, all you need to do is put $ in the Json Document attribute field. 2 other things I learned along the way and I have no better place to share them: 1) the range key in the processor is the sort key you defined when you created your table 2) if you have a + in your secret key Nifi has a hard time encoding it or something, and you won't be able to authenticate.
... View more