Member since
08-21-2024
5
Posts
2
Kudos Received
0
Solutions
09-18-2024
05:45 AM
Hi @SAMSAL it's actually more of case 1, so the exact scenario is... 1. a file is uploaded to a location 2. processor sees there is a new file and then initiates the 2 separate flows based on different sheets in a spreadsheet 3. after the 2 processes have run and generated their data, I'd like to merge that back together into a single JSON file Not sure how the fork enrich / join works so any light you can shed on that would be great. Alternatively, is there a better way to access data from a spreadsheet (2 different sheets, but the format and naming of the data is exactly the same) that I can use to then perform JOLT's on or something? I tried listing both sheets in the record reader but data 2 overwrites data 1 as they are exactly the same field / header names etc Thanks in advance for all the help!
... View more
09-18-2024
03:06 AM
1 Kudo
hey all I was just wondering what the best way is to merge 2 JSON flowfiles together. I have 2 processes which generate pretty much identical data but with a different source (so can’t do it as one process) and I’d like to merge them together into one JSON file if possible. Example data is something like... Process 1: [
{
"code": "ABC123",
"product": "My Product",
"prices": [
{
"valid_from": "20180913 09:00:08",
"name": "pricelist1",
"abc": "14.74"
},
{
"valid_from": "20180913 09:00:08",
"name": "pricelist2",
"abc": "10.70"
}
]
}
] Process 2: [
{
"code": "ABC123",
"product": "My Product",
"prices": [
{
"valid_from": "20180913 09:00:08",
"name": "pricelist1",
"xyz": "53.74"
},
{
"valid_from": "20180913 09:00:08",
"name": "pricelist2",
"xyz": "21.70"
}
]
}
] Desired Outcome: [
{
"code": "ABC123",
"product": "My Product",
"prices": [
{
"valid_from": "20180913 09:00:08",
"name": "pricelist1",
"abc": "14.74",
"xyz": "53.74"
},
{
"valid_from": "20180913 09:00:08",
"name": "pricelist2",
"abc": "10.70",
"xyz": "21.70"
}
]
}
] There is actually an array of products, although I’ve only shown 1 example above. Thanks in advance for any help
... View more
Labels:
- Labels:
-
Apache NiFi
09-02-2024
04:42 AM
1 Kudo
Hi all I was looking for some help regarding moving a NiFi Registry set up from one server to another. At the moment we set up a kind of "QA" system which was running well and so we have now created the production server if you like which we're going to use for all of our NiFi connections. The current registry is backed by a Git Repo using BitBucket which all works as expected. We have set up the new server, installed registry and connected the same repo and all seems to be fine in terms of the NiFi servers can connect and I can see registry on the new server. What I want to do now though, is move everything we had in the "QA" registry server, into the production server if this is possible (so that means the buckets, versions and flows etc) but JUST for the registry (we will then pull these fresh into the NiFi server(s)). I think I read somewhere that the information in registry is kinda "linked" to a registry server via a UUID or something but I'm not sure how that works (if not, my assumption would be that everything already in the repo should just work in the registry server but it doesn't show up). Is this the case? Is there a standard way to move something from one server to another, or a set of tools / commands that I can run to make sure that everything we have in the current registry server is in the new production server? Many thanks in advance everyone, very much appreciated! Craig
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
08-22-2024
12:46 AM
Hey @SAMSAL - thanks for the detailed reply, and yep you hit the nail on the head ha ha. I've used a few of those links already, and as you said I was close, but not quite there. I grasp the concept, it's just the syntax of everything that was throwing me off. That being said, the JSLT example you posted does seem to make more sense to me also, so that's definitely something I'll look into. I spoke to a guy called Paul Lakus in the NiFi slack channel and he basically gave me almost exactly the same solution that you have posted and so I managed to work out how it was being done, but still a huge learning curve for me heh. Anyway this now seems to be working as intended, which I'm extremely grateful for so thank you for your input and the steer towards JSLT - definitely something to look into. Thanks!!
... View more
08-21-2024
05:40 AM
HI all I've been using this community support a LOT lately as I try and get into NiFi and the ETL power it has, and I think it's incredible, but today I find myself instead asking for help, after spending far too long trying to work something out on my own 😕 I'm not too familiar with JOLT and so what I'm trying to do is take a data flow I have which is in the following format: [ { "CODE": "1234", "PRODUCT": "product 1", "Valid": null, "Zone 1": "1.00 USD", "Zone 2": "1.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null, }, { "CODE": "9876", "PRODUCT": "product 2", "Valid": null, "Zone 1": "18.00 USD", "Zone 2": "12.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null }, { "CODE": "3745", "PRODUCT": "product 3", "Valid": null, "Zone 1": "51.00 USD", "Zone 2": "21.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null } ] and try to convert it to something like this: [ { "code": "1234", "product": "product 1", "prices" : [ { "name" : "Zone 1", "price": "1.00 USD", "valid": null }, { "name" : "Zone 2", "price": "1.00 USD", "valid": null }, { "name" : "Zone 3", "price": null, "valid": null }, { "name" : "Zone 4", "price": null, "valid": null }, { "name" : "Zone 5", "price": null, "valid": null } ] }, { "code": "9876", "product": "product 2", "prices" : [ { "name" : "Zone 1", "price": "18.00 USD", "valid": null }, { "name" : "Zone 2", "price": "12.00 USD", "valid": null }, { "name" : "Zone 3", "price": null, "valid": null }, { "name" : "Zone 4", "price": null, "valid": null }, { "name" : "Zone 5", "price": null, "valid": null } ] }, { "code": "3745", "product": "product 3", "prices" : [ { "name" : "Zone 1", "price": "51.00 USD", "valid": null }, { "name" : "Zone 2", "price": "21.00 USD", "valid": null }, { "name" : "Zone 3", "price": null, "valid": null }, { "name" : "Zone 4", "price": null, "valid": null }, { "name" : "Zone 5", "price": null, "valid": null } ] } ] using my very limited knowledge of JOLT (and incredible search engine skills ha ha) I've managed to get something similar in the playground by trial and error and so I am now at a position like this: [ { "code": "1234", "product": "product 1", "price_lists": [ { "Zone 1": "1.00 USD" }, { "Zone 2": "1.00 USD" }, { "Zone 3": null }, { "Zone 4": null }, { "Zone 5": null } ] }, { "code": "9876", "product": "product 2", "price_lists": [ { "Zone 1": "18.00 USD" }, { "Zone 2": "12.00 USD" }, { "Zone 3": null }, { "Zone 4": null }, { "Zone 5": null } ] }, { "code": "3745", "product": "product 3", "price_lists": [ { "Zone 1": "51.00 USD" }, { "Zone 2": "21.00 USD" }, { "Zone 3": null }, { "Zone 4": null }, { "Zone 5": null } ] }, null ] and this is the spec I have used: [ { "operation": "shift", "spec": { "*": { "CODE": "[&1].code", "PRODUCT": "[&1].product", "Valid": "", "*": "[&1].price_lists[].&" } } } ] but sadly I think I have reached the end of my knowledge. I just can't seem to work out how to build the price_lists array in the format that I need it! I know we can reference hierarchy and also get key values as data using things like "$" and "@" but I simply can't get enough of an example to be able to make sense on how it all works. I may well be going about it the wrong way, but I think this is the right solution, I just don't have the knowledge to be able to achieve what I would like, so I'm reaching out to this amazing community and asking (nay begging) for help ha ha Thanks in advance all, this is an amazing place !
... View more
Labels:
- Labels:
-
Apache NiFi