@Surendra Shringi We can do this parsing inside NiFi by using Example:- Let's consider your csv file having n number of rows in it Surendra,24,"{"city":"Chennai","state":"TN","zipcode":"600345"}" Surendra,25,"{"city":"Chennai","state":"TN","zipcode":"609345"}" We need to split this file into individual flowfile having each record in one flowfile for splitting we need to use SplitText:- processor with below configs as Line Split Count 1 So if our input csv having 2 lines in it then split text processor will split the input file having 2 lines into 2 flowfiles having each line in one flowfile. Once we are having each record in one flowfile then we need to use ExtractText:- to extract the content of the flowfile using Extract text processor by adding new properties to the processor as below. Address_city "city":"(.*?)" Address_state "state":"(.*?)" Address_zipcode "zipcode":"(.*?)" Age ,(.*?), Name ^(.*?), So in this processor we are going to extract contents of flowfile and keep them as flowfile attributes by adding matching regex. To create and test regex click here. You need to change Maximum Buffer Size value (default is 1MB) based on your flowfile size. Replace Text Configs:- In the previous step we have extracted all the contents of flowfile based on the properties in Replace Text processor we are going to create a new csv file with comma delimiter(you can use any delimiter you want), By changing below properties and adding replacement value property as follows. Configs:- Search Value (?s)(^.*$) Replacement Value ${Name},${Age},${Address_city},${Address_state},${Address_zipcode} Maximum Buffer Size 1 MB Replacement Strategy Always Replace Evaluation Mode Entire text So the output of the replace text processor would be Surendra,24,Chennai,TN,24 Surendra,25,Chennai,TN,24 we have created a csv file without json message now but we are going to have 2 csv files(because our input data having 2 lines),if your input file having 1000 lines then we are going to end up with 1000 ourput csv files. If you don't want to create 2 output files and want them to merge into 1 output file then you need to use Merge Content Processor:- With the below configs, You need to change all the highlighted properties as per your requirements as per my configs shows Max bin age of 1 min so processor waits for 1 minute before merging all the queued flowfiles and merges them into 1 file. Delimiter strategy to Text(default is filename) because we need to have our contents of individual flowfile needs to add as newlines in the merged file, so we need to make use of Demarcator property as Shift+Enter(this property helps to add new contents to the newline). Output:- 1 file having both records in it Surendra,24,Chennai,TN,600345 Surendra,25,Chennai,TN,609345 I highly sugges you to refer below links to get familiar with all properties in merge content processor https://community.hortonworks.com/questions/149047/nifi-how-to-handle-with-mergecontent-processor.html https://community.hortonworks.com/questions/149047/nifi-how-to-handle-with-mergecontent-processor.html . I'm attaching the xml to the post you can save the xml and import to nifi and make changes to that accordingly.parse-file-nifi-159780.xml . If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors. ... View more

@Daniel Müller Can you create alias name for the aggregate column sum(menge) as menge then the processor will work. Change the SQL Query as Select sum(menge)menge from <table> ... View more

@Shakeel Ahmad You need to use InvokeHTTP processor to get response back from the API. InvokeHTTP Configs:- change the Remote URL property value to your url that gives back you required response. Change the Username,Password property values also. In this invoke http processor will get the JSON response and then you need to use Evaluate Json Path processor:- if you are going to extract only one attribute then you need to add new property that matches with the Json attribute. Example:- 1.If your json response is flattened as like below { "id": "1", "Certificate": "HCC" } Then the output of EvalJsonpath processor with above Configs would be HCC 2.If your json response is like below { "data": { "id": "1", "Certificate": "HCC" } } Configs of EvalJSON Processor:- As you can see above screenshot i have changed the Certificate Property value as my JSON response Certificate is inside the data list. so i have matched the Certificate attribute using $.data.Certificate which will result the output as HCC You need to change the Certificate property value as per your JSON Response. Once you get output from the EvaluateJSONPath processor the results are stored in a File, You can store the file using PutFile(or)PutHDFS...etc Flow:- 1.GenerateFlowFile Processor //Schedule to trigger InvokeHTTP processor 2.InvokeHTTP //Change the url,username,password property values and connect Response relation to EvalJsonPath 3.EvalJsonPath //add Certificate matching property value and connect matched relation to next processor if you are going to store the result into HDFS or Local then 4.PutHDFS(or)PutFile //to store the output file from EvalJsonPath to Local (or) HDFS ... View more

@dhieru singh I think just add capture all after $ as shown below screenshot will work. Search Value ^ $(.*) Replacement Value $1 Replacement Strategy Regex Replace Evaluation Mode Line-by-Line Replace Text Configs:- ... View more

@Kevin Vasko Yes we can do your case using NiFi Processors without using any external scripts. Here is what i tried:- First you need to extract the date from filename and keep it as attribute to the flowfile by using Update Attribute processor:- add new property date ${filename:substringAfter('_'):substringBefore('.')} Configs:- Then use Replace text processor to prepend the existing flowfile csv data with extracted date attribute Change the Replacement Value ${date} **keep space after ${date} in above. Maximum Buffer Size 1 MB //if your flowfile content is more than 1 MB then you need to change this value. Replacement Strategy Prepend Evaluation Mode Line-by-Line Configs:- So in this processor we are just prepending the data with date attribute that is extracted in Update attribute processor and we are going to prepend date value Line-by-Line evoluation mode. Then use another replace text processor to replace last , with double quotes " Search Value (.*),(.*),(.*), Replacement Value $1,"$2,$3" Maximum Buffer Size 1 MB //if your flowfile content is more than 1 MB then you need to change this value. Replacement Strategy Regex Replace Evaluation Mode Line-by-Line Configs:- So in this processor we are extracting all the data between last 2 commas and then replacing first group as is and enclosing group2 and group3 with double quotes and removing , at last. Output:- 2017-09-20 23:49:38.637,162929511757,$009389BF,36095,,,,,,,,,,"Failed to fit max attempts (1=>3), fit failing entirely (Fit Failure=True)" 2017-09-20 23:49:38.638,162929512814,$008EE9F6,-16777208,,,,,,,,,,"Command Measure, Targets complete - Elapsed: 76064 ms" I have attached the xml file you can use the same xml file and change the configs as per your needs. Flow:- Reference xml:- date-155661.xml Let me know if you have any issues..!! If the Answer helped to resolve your issue, Click on Accept button below to accept the answer, That would be great help to Community users to find solution quickly for these kind of errors. ... View more