Member since
07-29-2020
574
Posts
323
Kudos Received
176
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3556 | 12-20-2024 05:49 AM | |
| 3812 | 12-19-2024 08:33 PM | |
| 3602 | 12-19-2024 06:48 AM | |
| 2345 | 12-17-2024 12:56 PM | |
| 3091 | 12-16-2024 04:38 AM |
08-01-2023
11:35 PM
Good morning @cotopaul @SAMSAL First of all thank you very much for your time Ok Ok so I see that I missunderstood how QueryDatabaseTable is "triggered". @SAMSAL I knew about those configuration but as I said I thought this processor only do the query when a new record is inserted in the database taking reference the maximum column value. Makes sense now the postgresql log...the querys executed every second since my RunSchedule is on default (0). What I use minifi for is to load sales from a supermarket then send it via API (and others transformations), so I need this to be in real time as soon as the client paid. I guess I have to define Run Schedule every few seconds. @cotopaul I will take a look at my DBCPConnectionPool, this info it´s helpfull Thank you all.
... View more
08-01-2023
06:54 AM
1 Kudo
Hi @MWM , What you are describing is a classical data enrichment pattern that can be achieved using ForkEnrichment & JoinEnrichment processors. For more information on this please refer to : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html Based on your scenario the "SQL" strategy of the JoinEnrichment will work the best for you since you can select the main fields (system, name, surname, phone, mail) from the original flowfile data and select proffesion and department from enrichment result: SELECT o.system, o.name, e.surname,o.phone, o.mail, e.profession, e.department FROM original o LEFT OUTER JOIN enrichment e ON o.name= e.name Since you are splitting the CSV and enrich per record then you can just join by name. If you have an API where you can get a collection of user information then you dont have to split and you can do the enrichment on multiple records from the CSV vs. returned records from the API json output , however be aware that if you have large data set this strategy "... can lead to heap exhaustion and cause stability problems or OutOfMemoryErrors to occur". Please review the link above to see how this can be mitigated. If you find this is helpful please accept solution. Thanks
... View more
07-31-2023
01:42 PM
Hi @nict , Not sure how you want to add your headings (firstDetails, secondDetails...etc.) specially secondDetails is not found anywhere in your jolt spec. If you are just looking to associate different records (test1, test2, test3...etc.) to different header based on the value of "Name" where you have to list each Name value explicitly, then something like this will work: [
{
"operation": "shift",
"spec": {
"dataPackage": {
"datatest": {
"field": {
"*": {
"Name": {
"test1": {
"@(2,content)": "firstDetails.test1"
},
"test2": {
"@(2,content)": "firstDetails.test2"
},
"test3": {
"@(2,content)": "secondDetails.test3"
}
}
}
}
}
}
}
}
] You can add more values\header as needed. However, if you are looking to create header dynamically based on the Name values then you have to provide more info on how you would associate records with header. If that helps please accept solution. Thanks
... View more
07-30-2023
04:45 PM
I see, it work! Thanks for your information
... View more
07-26-2023
06:51 AM
@SAMSAL This looks like a plausible solution. Let me try it out.
... View more
07-25-2023
01:37 PM
@Diga In a scenario like this, the first thing to ask yourself is "How would I do this outside of NiFi?". If you were to use curl from command line, how would you handle authentication with the Polarion ALM rest-api endpoints? The answer there would help with how you would handle it within NiFi and assist in getting a better response in the community. @SAMSAL provided some great suggestions of possible methods. Matt
... View more
07-24-2023
09:11 PM
thank you so much! i'm moving forward again!
... View more
07-21-2023
07:49 AM
Hi @CommanderLaus , If you do anything manually with the processor you are trying to change its Running State through the API, then yes the revision will be updated. In my case what I do is basically the first API InvokeHttp is to get the latest revision using the following GET API: https://localhost:8443/nifi-api/processors/${pipeline.start.processor.id} This will give the following response where you can parse the latest "version" value using something like EvaluateJsonPath processor and store the value in an attribute: {
"revision": {
"clientId": "value",
"version": 0,
"lastModifier": "value"
},
"id": "value",
"uri": "value",
"position": {…},
"permissions": {…},
"bulletins": [{…}],
"disconnectedNodeAcknowledged": true,
"component": {…},
"inputRequirement": "value",
"status": {…},
"operatePermissions": {…}
} Then use extracted version attribute to build the next API post Body , for example in my case I use ReplaceText processor to generate the post body : {
"revision": {
"clientId": "${pipeline.start.processor.id}",
"version": ${processorVersion}
},
"state": "...",
"disconnectedNodeAcknowledged": true
} Hope that helps.
... View more
07-21-2023
04:25 AM
Thanks @SAMSAL for recommending @MattWho
... View more