Member since
05-20-2022
66
Posts
6
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1159 | 07-17-2023 11:25 PM | |
1150 | 04-17-2023 02:29 PM | |
5462 | 02-15-2023 08:47 PM | |
891 | 02-08-2023 06:02 PM | |
4344 | 10-17-2022 11:48 AM |
01-31-2023
09:56 AM
Great questions @SAMSAL I would like to do something like storing the JSON in a parameter contexts variable then using a Lookup service to retrieve the corresponding record. I think of it as an in-memory table that I can use to perform inner joins with flow files. Alternatively, I could read the table from a remote source (using InvokeHTTP) and then load it into a DistributedMapCacheLookupService, but I'm not familiar with this approach so I'd have to do some research. I appreciate your time. Thank you.
... View more
01-30-2023
09:59 PM
I have a flow file that I want to enrich with the contents from a list of JSON values based on the "id". Basically perform an inner join of the flow file with this look-up data set. However, I need the look-up dataset to be in memory, then using the "id", query the look-up dataset and append the results to the current flow file. Here is an example: Incoming flow file: {
"id": "abc123",
"fname": "The",
"lname": "Rock"
} Contents of the look-up data set: [
{
"id": "abc123",
"dob": "03/09/1977",
"phone": "987-654-0001"
},
{
"id": "def765",
"dob": "04/08/1976",
"phone": "789-654-0001"
},
{
"id": "hij765",
"dob": "05/06/1975",
"phone": "456-654-0001"
}
] Enriched flow file: {
"id": "abc123",
"fname": "The",
"lname": "Rock",
"dob": "03/09/1977",
"phone": "987-654-0001"
} I need to be able to look up the correct record in the look-up dataset based on the "id" then append the values to the current flow file. The key here is that I need the look-up dataset to reside in memory (can't be a file or a database) Thanks for reading and look forward to hearing back with ideas.
... View more
Labels:
- Labels:
-
Apache NiFi
10-17-2022
11:48 AM
@MarioFRS Ah, you are correct. I overlooked that additional nested Product. That complicates things because the upper Product can be an array or not any array, which itself contains another ProductDetails.Product which also may or may not be an array. Ugh! To solve this I had to resort to chaining multiple transforms to account for all the possible formats that can exist. I may not have all the possible permutations you'll have, but you can follow the basic structure of these transforms and create new ones with additional levels if necessary. It is important to note that the inner-most portions of the nested data need transformed first, so the order of the transforms starts with the inner-most data and ends with the outer-most data. Keep that in mind if you need to add more transforms to account for additional permutations of the data. Give this a try and let me know how it works for you. [
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"*": {
"*": {
"*": {
"*": {
"*": {
"Product": "MANY"
}
}
}
}
}
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"*": {
"*": {
"*": {
"*": {
"Product": "MANY"
}
}
}
}
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"*": {
"*": {
"Product": "MANY"
}
}
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"*": {
"Product": "MANY"
}
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"ClassSummary": "MANY"
}
}
}
},
{
"operation": "cardinality",
"spec": {
"*": {
"ClassSummary": "MANY"
}
}
}
]
... View more
10-14-2022
04:23 PM
Hey @MarioFRS, Just following up to make sure things are good to go. If you are all set then please accept the solution to ensure it helps others in the future.
... View more
10-13-2022
02:16 PM
Based on this screen shot it appears you are using the wrong Jolt Transform. Try using the newest version that I sent earlier this morning. I've also attached screen shots which show the Jolt works with your data. I trimmed your data up a little for brevity purposes, but the structure is still the same.
... View more
10-13-2022
12:26 PM
Hmm. I tried all the use cases (single object, array, no array) and this transform works with all the example data that you've posted. Can you send a screen shot of what you mean; perhaps I don't understand the problem.
... View more
10-13-2022
08:52 AM
Thanks @SAMSAL, I was tracking that previous post but since it was so old I was hoping newer versions of NiFi could address this. This is such a simple thing with XML and the EvaluateXPath processor, I am just surprised that we can't do something similar with JSON. Oh well, 2 processors it is.
... View more
10-13-2022
08:33 AM
This one works. Give it a try... [
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"ClassSummary": "MANY",
"*": {
"*": {
"Product": "MANY"
}
}
}
}
}
}
]
... View more
10-13-2022
01:40 AM
"Product" is nested several layers deep so the first thing you need to do is match the wildcard (*) character to match the depth of where "Product" is located. As a simplified example, I trimmed and flattened your JSON data down a little, and the transform works as expected. Where I ran into a challenge was with the "ClassSummary" array, because I don't know how to account for it in the structure. Hopefully this gives you something to work with, and using this web site you get some ideas on how to take it further. I'm very curious to see the result. https://jolt-demo.appspot.com/#ritwickgupta [
{
"operation": "cardinality",
"spec": {
"*": {
"*": {
"Product": "MANY"
}
}
}
}
] {
"costc": 9638,
"sum_amount": 543,
"Classes": {
"class": 1102,
"sum_amount": 117,
"sum_invitation": 0,
"ProductDetails": {
"Product": {
"id": 7992160,
"artnr": 32212,
"sum_amount": 16,
"sum_invitation": 0,
"value": null
}
}
},
"date": "2022-10-06"
}
... View more
10-12-2022
04:27 PM
I need to use wildcard syntax in a JSON path to return the color of the bike as a scalar value. However, the problem is that when I use wildcard syntax with the EvaluateJSONPath processor it always returns an array. For example This path returns a scalar value: $.store.bicycle.color ==> red This path returns an array: $..bicycle.color ==> ['red'] Is it possible to use the EvaluateJSONPath processor with wildcards and get a scalar value in return? Or, is it possible to extract the value from the array without using an additional processor? {
"store": {
"bicycle": {
"color": "red",
"price": 19.95
}
},
"expensive": 10
}
... View more
Labels:
- Labels:
-
Apache NiFi