Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SQL to NoSQL: How to re-write an output JSON to the DynamoDB format?

SQL to NoSQL: How to re-write an output JSON to the DynamoDB format?

New Contributor

Hello, everyone!

I'm writing a process that extracts ~1.5 millions rows from a SQL Stored Procedure to one single JSON file, and put it in a DynamoDB table using AWS CLI command.


My currently workflow below:

108992-screen-shot-2019-05-27-at-160654.png


I'm using the ExecuteSQLRecord to extract the data from the SQL Stored Procedure. My JSON output looks exactly like this:

[ {
  "CUSTOMER_ID" : "11111111111",
  "TRANSACTION_TYPE" : "G",
  "ID_TRANSACTION" : 3,
  "ACCOUNT_ID" : 111111111,
  "TRANSACTION_VALUE" : "100.00"
}, {
  "CUSTOMER_ID" : "22222222222",
  "TRANSACTION_TYPE" : "G",
  "ID_TRANSACTION" : 10,
  "ACCOUNT_ID" : 222222222,
  "TRANSACTION_VALUE" : "1000.00"
}, {
  "CUSTOMER_ID" : "33333333333",
  "TRANSACTION_TYPE" : "R",
  "ID_TRANSACTION" : 8,
  "ACCOUNT_ID" : 333333333,
  "TRANSACTION_VALUE" : "3000.00"
}
 ]

Each row from the resultset is an item in the JSON file.


After that, I use the PutFile to save the JSON file in my local machine, then I'm going to use the ExecuteStreamCommand to execute the AWS CLI command.


The question is: How can I format my JSON output file to a JSON format file that respects the DynamoDB rules (with the RequestItems and TableName attributes)?


Any help will be appreciated

Thanks!

1 REPLY 1
Highlighted

Re: SQL to NoSQL: How to re-write an output JSON to the DynamoDB format?

The above was originally posted in the Community Help Track. On Tue May 28 19:45 UTC 2019, a member of the HCC moderation staff moved it to the Data Ingestion & Streaming track. The Community Help Track is intended for questions about using the HCC site itself.

Bill Brooks, Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Don't have an account?
Coming from Hortonworks? Activate your account here