Member since
08-18-2019
56
Posts
11
Kudos Received
18
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
655 | 11-15-2023 05:38 AM | |
2378 | 07-12-2023 05:38 AM | |
710 | 01-10-2023 01:50 AM | |
995 | 12-06-2022 12:08 AM | |
3860 | 12-02-2022 12:55 AM |
10-08-2022
08:23 AM
1 Kudo
Hey @Fredi , I believe the answer for your problem is the processor UpdateRecord. Update record allows you to directly manipulate the fields in your file content. You add dynamic properties to the processor where the key of the property is /<field> (so in your case, '/api_value'), and in the value of this dynamic property you can write down some logic to determine what value to insert into api_value. In the processor, there is a field called "Replacement Value Strategy", which defines how the value of the property will be read. If you set this to "Record Path Value", it means you can now give a path to a different field in your file (url_value!) - I can't test this right now because I'm not at my office, but I'm not entirely sure whether you can manipulate the result after giving a record path (to extract the api_value from the evaluated url_value). Regardless, I'm just about 100% sure this can be done with two processors - One EvaluateJsonPath to extract the url_value into an attribute, then UpdateRecord that uses the 'Literal Value' replacement strategy - with this strategy, you can just add a property with key '/api_value' and value '${url_value}' (or whatever attribute name you gave to the extracted url_value) and once you can access url_value with the expression language (via ${url_value}) you can use all the available functions to manipulate expression language variables. Here's an article with a couple of examples on UpdateRecord: https://community.cloudera.com/t5/Community-Articles/Update-the-Contents-of-FlowFile-by-using-UpdateRecord/ta-p/248267 (I noticed in the article they used some recordPath related functions like "replaceRegex", so I believe there might be a way to use these and then limit the entire issue to just one UpdateRecord processor! Sadly I'm not too familiar with these myself and this was the first time I've seen them) And here's the expression language documentation: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html You can see there are lots of useful functions to extract your api_value once you have ${url_value} as an attribute variable, for example "substring", "find"/"replace", "ifElse", etc. all of which you can try and use to ensure only the api_value is left in the end. Hope this helps! I'm sure using ReplaceText and possibly JoltTransform could provide alternate solutions to the issue, however I believe UpdateRecord is the cleanest solution for this and truly makes use of the processor's abilities. If you struggle to use it correctly, you can reply with an example json and expected output and I'll try to write down the flow when I have time.
... View more
09-26-2022
07:08 AM
Hello, you can configure that just 1 Flowfile will be handled inside of a specific ProcessorGroup. There is the config option on PG which is called: Process Group FlowFile Concurrency for that you can set the value: Single FlowFile per Node After PutKudu you will destroy the FlowFile or route it out from ProcessorGroup, then the next FlowFile will be released to enter to ProcessorGroup In your case the Flow would look like: ListFile Processor -> ProcessorGroup (handels fetchFile, data Transformation and putKudu)
... View more
02-18-2022
12:33 AM
what if the value is empty. How to set null value instead of empty string ?
... View more
01-21-2021
07:06 AM
@Lallagreta Make sure you do not have any line returns in the values for your dynamic properties added in the UpdateAttribute processor. When you click on the value field for each property you should not see a line "2". For example: Above would result in the value assigned to the FlowFile Attribute having a line return. If this is the case, edit the properties value(s) to remove the line returns so you only see one line (1). Hope this helps, Matt
... View more
01-13-2021
07:51 AM
@FaerballertThanks again. It turned out that I just should ignore the question marks in the SQL statement. Everything works for now.
... View more
01-13-2021
04:47 AM
Did you already think about to transform your csv to json and then rebuild it with JOLT?
... View more
01-08-2021
05:23 AM
2 Kudos
@murali2425 The solution you are looking for is QueryRecord configured with a CSV Record Reader and Record Writer. You also have UpdateRecord and ConvertRecord which can use the Readers/Writers. This method is preferred over splitting the file and adds some nice functionality. This method allows you to provide a schema for both the inbound csv (reader) and the downstream csv (writer). Using QueryRecord you should be able to split the file, and set attribute of filename set to column1. At the end of the flow you should be able to leverage that filename attribute to resave the new file. You can find some specific examples and configuration screen shots here: https://community.cloudera.com/t5/Community-Articles/Running-SQL-on-FlowFiles-using-QueryRecord-Processor-Apache/ta-p/246671 If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post. Thanks, Steven
... View more
01-07-2021
04:23 AM
Hi Faerballert Thanks for helping . I am able to get entire content into attribute . I have one doubt that i have attribute name called "JSONValue" so that extracted value should be in this name but the flowfile content saves into the attribute "JSONValue.0 " as supposed to create value in JSONValue attribute . Please see screen-print for the same . I had to enable "Include Capture Group 0" as True otherwise the Regex not accepting and expecting value 0 to40 other than false.
... View more
01-03-2021
06:33 AM
Sorry, cant visualize it, but your flow can be look like this: QueryDatabaseTableRecord -> Wait ->PutS3Object
-> FetchFile -> EvaluateJsonPath -> Notify You can overhand the attribute from JSON to the waiting FlowFile. If you need help with configuration or other stuff for this, just text me.
... View more
12-04-2020
07:04 AM
try this "createdon" : {
"$date" : "putInHereDate"
} instead of "createdon": ISODate("2017-03-03") you can solve this also via JOLT
... View more