Member since
07-29-2020
574
Posts
320
Kudos Received
175
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
240 | 12-20-2024 05:49 AM | |
274 | 12-19-2024 08:33 PM | |
286 | 12-19-2024 06:48 AM | |
237 | 12-17-2024 12:56 PM | |
223 | 12-16-2024 04:38 AM |
11-22-2024
03:12 AM
3 Kudos
Hi, It seems like the Url you have in the invokehttp is invalid compared to the one in postman: invokehttp: https://tolingo-portal-test.s.xtrf.eu/home-api/quotes/ZQ6BPIPYHVE2FCVQ4HASNRIU3I/status Postman: https://tolingo-portal-test.s.xtrf.eu/home-api/v2/quotes/ZQ6BPIPYHVE2FCVQ4HASNRIU3I/status It seems like you are missing the v2 in invokehttp. When I try the correct url with the following configuration it works but I dont have access ofcourse 🙂
... View more
11-21-2024
01:41 PM
Hi, Can you elaborate more about the API call and what method its using? if you also can provide how the API works using postman or curl that would be helpful too. Just FYI, if you are trying to send json body with Get method its not going to work using this processor and you have to write custom code using ExecuteScript to work around it.
... View more
11-20-2024
09:03 AM
1 Kudo
Hi, Im unable to replicate the error . Can you provide more details about your flow with the processor configurations. here is what I tried and it worked: GenerateFlowFile: EvaluateXPath: Output flowfile attributes:
... View more
11-19-2024
09:31 AM
1 Kudo
Hi, Can you provide sample of this data highlighting what the problem and how do you expect to solve? Also what kind of processor\service are you trying to use to parse this data?
... View more
10-31-2024
04:55 PM
2 Kudos
Hi @Syed0000 , Sorry for the delay. You json is quite complex and very nested which make jolt very hard to write. You probably know your data very well and my recommendation before writing jolt for such Json is that you try to simplify it first by stripping un need blocks and\or flattening the structure where the transformation is not going to be on the same nested level. I honestly tried but when the jolt got so nested it got harder to reference the upper fields and upper array indexes to maintain the same array position and then maintain the required grouping of fields, and may be that is the draw back of using jolt spec in these scenarios (without prior simplification of course :). This leads me to the second option, which I think I have mentioned to you in prior post regarding using JSLT Transformation instead. JSLT is better option in these cases because you can traverse the structure much easier , and since you have some condition on how to set fields and values this also would be easier to achieve with some expression language like if-else . For example if we take the transformation to create catalogCondition strucutre which seems to the most complex, here is how the jstl looks like: let paths ={
"paths":[for(.tiers)
{
for(.conditions)
"catalogCondition":
{
"category":
[for(.conditionsMerch)
{
"include": if(.excludeInd==0) "yes" else "no",
"categoryList":[.webCategoryId]
}
if(.merchandiseLevel!=5 and not(.keyItemNum))]
}
+
{
"products" : [for(.conditionsMerch)
{
"include": if(.excludeInd==0) "yes" else "no",
"productList":[.keyItemNum]
}
if(not(.webCategoryId))]
}
}
]
}
{
"pricePaths": $paths
} I know this might look a little intimidating at beginning but its not near as bad as when you try to do it with jolt. I understand that jstl is bit of learning curve but I believe it would save you tons of time long term specially when dealing with long, complex and nested Json transformation. I know this is not the answer you were looking for but hopefully this can help you when facing other json transformation challenges in the future. Good luck.
... View more
10-30-2024
04:41 AM
1 Kudo
Hi @fisblack , Welcome to the community. Your transformation is kind of odd. Usually transformation is driven by existing values or structure even when generating new fields with defaults. I have never seen a case where newly generated values drive transformation of existing ones. I assume you are getting the list of courseSubCategory from somewhere and the script is rather dynamic utilizing some expression language & attributes. if you provide some context to how the list is coming about and what are you trying to achieve , maybe we can provide you with more accurate solution. There are multiple ways of solving your problem as is - assuming you always want to replicate an existing field(s) against newly generated 3 elements array with default values of 1,2&3- , here is one of them: [
{
"operation": "shift",
"spec": {
"*": "existing.&",
"#1|#2|#3": "courseSubCategory"
}
},
{
"operation": "shift",
"spec": {
"courseSubCategory": {
"*": {
"@(2,existing)": "[&1]",
"@": "[&1].courseSubCategory"
}
}
}
}
] If you want to make this more dynamic in terms of number of "courseSubCatgory", you can construct the array assignment string "#1|#2|#3" in some upstream processors and store the value as flow attribute (let says its called the same) , then you can make it more dynamic by using the flowfile attribute where the assignment of the courseSubCategory in the first shift will look like this: "${courseSubCategory}": "courseSubCategory" Hope that helps. If it does help, please accept the solution. Thanks
... View more
10-29-2024
07:51 PM
1 Kudo
Hi @Syed0000 , Do you mind posting what the expected output looks like. Also please provide description of what are you trying to achieve with the transformation?
... View more
10-29-2024
07:01 AM
1 Kudo
Hi @AnikaBaykus , I dont think this is going to work through InvokeHttp. I recommend you use custom Groovy code through Executescript or using Curl through ExecuteStreamCommand. Here are some links that can help you getting started: Curl via StreamCommand: https://community.cloudera.com/t5/Support-Questions/NiFi-Execute-Stream-Command-Curl-Call/m-p/368127 API vs Groovy script: https://stackoverflow.com/questions/57242452/in-nifi-how-to-call-a-rest-service-using-groovy-script ExecuteScript tutorial: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-1/ta-p/248922 https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-2/ta-p/249018 https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148 If that helps please accept the solution. Thanks
... View more
10-29-2024
03:05 AM
1 Kudo
Hi @rturn , Its an interesting case indeed. I'm not sure if this is intended by design or if its an actual bug. The fact that this happens only after the flowfiles exit the process group lead me to believe that whatever you are doing there it's indirectly causing this error. Explanation: - Typically when you get such an error , it means that you are trying pass number of value arguments to a query that doesnt have a matching number of column placeholders. You query doesnt list columns placeholders (?) hence the value 0 . The question is where is the 1 argument is coming from? - If you read the PutSQL processor description it allows for a value argument placeholder using the question mark character (?) provided argument value\type tuple in the form of sql.args.N.type & sql.args.N.value as flowfile attributes, where N stands for the position of each argument placeholder in the sql command. - Given your data flow, it seems like you are doing some insert sql commands in the upstream process group while utilizing argument value placeholders. - When doing the Merge (regardless how you set the Attribute Strategy) the merged output seem to keep those sql argument type & value attributes that are common (name & value) across the merged flowfiles. - The PutSQL behavior (this is where its a potential bug) seem to always pass any provided sql argument regardless if the actual query has placeholders or not. When the JDBC tries to execute the query it sees you are providing arguments but there is no placeholders and therefore it throws this exception. Mitigation: 1- One way to mitigate this issue is by adding an UpdateAttribute before the final PutSQL to delete all sql placeholder arguments with the following Regix in the "Delete Attribute Expression" property: sql\.args\.[0-9]+\.(type|value) 2- Another option to avoid passing attributes that might indirectly cause unintended consequences to downstream processors through using Process Group Concurrency and merge, as an alternative you can consider something like Wait-Notify pattern, however this will require a redesign of the dataflow. @MattWho @stevenmatison , Do you think this qualifies to be reported as a bug? Steps to Reproduce: 1- Create starting processor like GetFile or GenerateFlowFile 2- Add UpdateAttribute processor to create the following attributes: 3- Add a PutSQL processor that executes any arbitrary crud statement that doesnt have any placeholders. Resolution: Check the sql command for any placeholder and if there is none then dont pass arguments. If that helps please accept the solution. Thanks
... View more
10-25-2024
07:54 PM
2 Kudos
@rturn, Do you mind posting screenshot of your data flow and the configuration of the main processors? It doesnt need to be the exact thing if you can simplify so that you are able to replicate the error that would be fine. This will help us better troubleshoot the issue and replicate it to our environment if needed to see what its going on.
... View more