Member since
05-27-2019
4
Posts
0
Kudos Received
0
Solutions
06-05-2019
06:54 PM
Hello, Everyone!
I have been trying to use the PutDynamoDB processor, and I almost there. I'm able to run the ExecuteSQLRecord to get my data and store in a JSON file. The data is flowing, but I'm getting the following error in the log:
PutDynamoDB[id=28e4e8cf-016b-1000-a35c-276514f81580] Hash key value '' is required for flow file StandardFlowFileRecord[uuid=0ce692ff-d784-4882-a565-f5425479d316,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1559758360111-4, container=default, section=4], offset=0, length=275868738],offset=0,name=0ce692ff-d784-4882-a565-f5425479d316.json,size=275868738]
I understand that, is looking for the Hash Key Value, but I not sure how do I pass it. In the PutDynamoDB properties tab, NiFi sets
${dynamodb.item.hash.key.value}
as default value. But looks like this isn't the right way to do it.
Can I get some guidance on this?
My workflow:
The JSON I'm trying to put in DynamoDB looks like this:
[ {
"CUSTOMER_ID" : "11111111111",
"TRANSACTION_TYPE" : "G",
"ID_TRANSACTION" : 3,
"ACCOUNT_ID" : 111111111,
"TRANSACTION_VALUE" : "100.00"
}, {
"CUSTOMER_ID" : "22222222222",
"TRANSACTION_TYPE" : "G",
"ID_TRANSACTION" : 10,
"ACCOUNT_ID" : 222222222,
"TRANSACTION_VALUE" : "1000.00"
}, {
"CUSTOMER_ID" : "33333333333",
"TRANSACTION_TYPE" : "R",
"ID_TRANSACTION" : 8,
"ACCOUNT_ID" : 333333333,
"TRANSACTION_VALUE" : "3000.00"
}
]
Any help will be appreciated
Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
05-30-2019
01:28 PM
Hello, Alena! How are you? I have some doubts too about build a workflow to move data from SQL to NoSQL. Did you manage to solve this?
... View more
05-28-2019
07:45 PM
Hello, everyone! I'm writing a process that extracts ~1.5 millions rows from a SQL Stored Procedure to one single JSON file, and put it in a DynamoDB table using AWS CLI command. My currently workflow below: I'm using the ExecuteSQLRecord to extract the data from the SQL Stored Procedure. My JSON output looks exactly like this: [ {
"CUSTOMER_ID" : "11111111111",
"TRANSACTION_TYPE" : "G",
"ID_TRANSACTION" : 3,
"ACCOUNT_ID" : 111111111,
"TRANSACTION_VALUE" : "100.00"
}, {
"CUSTOMER_ID" : "22222222222",
"TRANSACTION_TYPE" : "G",
"ID_TRANSACTION" : 10,
"ACCOUNT_ID" : 222222222,
"TRANSACTION_VALUE" : "1000.00"
}, {
"CUSTOMER_ID" : "33333333333",
"TRANSACTION_TYPE" : "R",
"ID_TRANSACTION" : 8,
"ACCOUNT_ID" : 333333333,
"TRANSACTION_VALUE" : "3000.00"
}
] Each row from the resultset is an item in the JSON file. After that, I use the PutFile to save the JSON file in my local machine, then I'm going to use the ExecuteStreamCommand to execute the AWS CLI command. The question is: How can I format my JSON output file to a JSON format file that respects the DynamoDB rules (with the RequestItems and TableName attributes)? Any help will be appreciated Thanks!
... View more