Created 11-28-2017 02:10 PM
@Timothy Spann,@matt burgess : We are using Nifi to Extract Data and prepare staging Layer in Redshift but for Denorm layer we need to use a different tool Matillion for Transformations on Data like Joins etc. Could anyone suggest if we can do Transformations in Nifi itself and send data to denorm directly after extracting from sources in nifi.
Please Help!!
Created 11-29-2017 01:17 AM
Yes most transformations can be done in NiFi like changing types, dropping columns, changing formats, changing from JSON to AVRO or CSV to JSON. Lots of transformations.
Created 11-29-2017 05:38 AM
I read your answer on joining tables in Nifi at this post
but I am not able to use Merge Content to get a desirable wide row.
I have two sources which outputs json records: record1 has two columns policynumber & xmljoinkey(hash value of a xml) & other json has two columns: id & xmljoinkey(hash value of a xml). I want to join both the jsons on xmljoinkey and output a single json that has three fields xmljoinkey,policynumber,id.
but i am getting output in this manner:
{ "policynumber": "WC2494P2017", "xmljoinkey": "c1b40e1c4c611a4e5ede35176ab3abfdd0f45eb5" }
{ "policynumber": "WC2494P2017", "xmljoinkey": "c1b40e1c4c611a4e5ede35176ab3abfdd0f45eb5" }
{ "policynumber": "WC2494P2017", "xmljoinkey": "c1b40e1c4c611a4e5ede35176ab3abfdd0f45eb5" }
{ "xmljoinkey": "c1b40e1c4c611a4e5ede35176ab3abfdd0f45eb5", "transactionid": "1234" }
{ "xmljoinkey": "c1b40e1c4c611a4e5ede35176ab3abfdd0f45eb5", "transactionid": "5647" }
Could you please suggest some approach on this?
Thanks in advance!!
Created 11-29-2017 10:00 AM
I read the answer by Bryan Bende for Joins in nifi: https://community.hortonworks.com/questions/118424/how-to-merge-2-flow-file-content-coming-from-2-di...
so i think i might not be able to do this in Nifi , still if anyone finds any way doing this, please share.
Thanks a lot :-)
Created 07-16-2020 11:05 AM
@srijitachaturve Based on my experience of evaluating ETL tools, I have found the below tools to be helpful while joining data across tables.