Member since
06-19-2017
62
Posts
1
Kudos Received
7
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2766 | 03-17-2022 10:37 AM | |
1922 | 12-10-2021 04:25 AM | |
2486 | 08-18-2021 02:20 PM | |
6699 | 07-16-2021 08:41 AM | |
1234 | 07-13-2021 07:03 AM |
01-08-2021
09:14 PM
Hi All, I am trying to convert input JSON to Output JSON using Jolt Transformation . My flow," AttributestoJSON" gives some count of flow files and after that i am using "Mergecontent" to merge the flowflies into single flowfile JSON .The single flowfile json passed to jolt transformation. Input flow file JSON {"filename":"test1.csv","message":"HiveSQL query executed Sucessfull!","status":"Success"} {"filename":"test2.csv","message":"HiveSQL query executed Sucessfull!","status":"Success"} {"filename":"test3.csv","message":"HiveSQL query executed Sucessfull!","status":"Success"} Output JSON should be : { "filename": "${main_file}", "ex_status": "${Overall_status}", "error": "${final_message}", "fl_status": [ { "file_name": "test1.csv.", "status": "Success", "message": "HiveSQL query executed Sucessfull!" }, { "file_name": "test2.csv", "status": "Success", "message": "HiveSQL query executed Sucessfull!" }, { "file_name": "test3.csv", "status": "failed", "message": "HiveSQL query executed Sucessfull!" } ] } Here main_file,Overall_status and final_message values , i am getting as flowfile attributes and will include the vaues for that from the attributes . I am exploring JSON specification and did not get the desigred result yet . Please help with Jolt specification for above output JSON . Thanks @Nifi@Jolt
... View more
Labels:
- Labels:
-
Apache NiFi
01-07-2021
04:23 AM
Hi Faerballert Thanks for helping . I am able to get entire content into attribute . I have one doubt that i have attribute name called "JSONValue" so that extracted value should be in this name but the flowfile content saves into the attribute "JSONValue.0 " as supposed to create value in JSONValue attribute . Please see screen-print for the same . I had to enable "Include Capture Group 0" as True otherwise the Regex not accepting and expecting value 0 to40 other than false.
... View more
01-06-2021
08:55 AM
Hi All, I have nifi flow fthat fetches one json file and using the extractText processor, i want to store entire flowfile content(assume small json file) into one flowfile attribute. But (.*) is not extracting whole json and instead extracting only { . Please help with Java regular expression to extract full json afnd also share if any Java regular expression documentation which i can refer for Nifi extracttext processor. I having json file which contains format like { "filename1":"insert into filename1", "filename2":"insert into filename2", "filename3":"insert into filename3" } I will retrieve the filenames in generic way/dynamic using JsonPath(${filename}) in updateattribute . Please assist to extract whole json content into one flow file attribute . Thanks @Nifi@NifiDev
... View more
Labels:
- Labels:
-
Apache NiFi
01-06-2021
08:18 AM
@MattWho This is Amazing and solution working well.
... View more
01-04-2021
03:29 AM
Hi ,
@MattWho
Thanks a lot for clarifying and provide solution for Merging contents . I have implemented as per your guidelines and the merge content working fine But i am unable to update overall status of file if it is error and message as well. Please find my status flow in detail.
1.Your flow starts with a single zip file? --- I get meta file which contains .zip file to be fetched . for exmaple whosale.meta file which contain .zip file having 3 or 4 csv files .
2. You unzip that file to produce numerous output FlowFiles? --- Yes , I unzip the file .
3. You use load balanced connections to distribute all the produced FlowFiles across all nodes in your cluster? --- Yes , i do enable round robin load balancing for the flow files which was unzip in step 2 and push the flowfiles into HDFS .
4. Then you modify content of FlowFile using convertAttribitesToJson processor (Destination=flowfile-content)? Looks like you route the "success" relationship twice from this processor which means you have cloned your FlowFiles. Why? --- Initially i was trying convert attributes into JSON flowfile. I have to capature success and failure status of the processor .So i have avoided not clone the success relationship twice.
6. Where do the "failure" FlowFiles get introduced in to this dataflow? --- If i encounter "failure" in process of pushing files to HDFS like permission denied or folder not available . Then i will add attribute with some custom messages and send to merge content . And i have also dirceting success status with some custom message of the final process success status in attribute to replacetext followed with Mergecontent .
Status flow
HDFSError attribute :
HDFS Successattribute :
After merge content i do use Replacetext processor as suggested to add overall status of the file but somehow the status (error) and message is not captured .
ReplaceTextProc to construct over all status :
The final file will contain like below
Ingest_flow.zip : Success : test1.csv : Success : File artikelgroep_info.csv pused to PreRAW Zone test2.csv : Success : File demo.csv pused to PreRAW Zone test3.csv : Success : File work11.csv pused to PreRAW Zone
i could not populate over all message AND if any file error then i could not capture status " Failed" and with error in overall status of the .zip file .Could you please help on this
Thanks
... View more
12-29-2020
02:25 AM
Hi, I am trying to extract the single attribute(dynamic attribute passing) value from a JSON file or TEXT file . The config file format looks like below { "product.csv" :"INSERT INTO Product", "Wholesale.csv" :"INSERT INTO Wholesale" } The above JSON can evolve based new sources files and can added many file names with respective hive queries . I have config file at shareserver and able to retrieve the into middle of the nifi flow as per the attached screen-print . I able to retrieve the attribute values using Evaluate JSON path with fixed number of attributes . But the attribute value can be changed and i can't configure the same attribute name in Evaluate JSON path processor . I understand that getting attribute value by dynamically passing attribute names not possible in nifi processor like JSON to Attributes . https://community.cloudera.com/t5/Support-Questions/Nifi-Dynamic-Attribute-Creation/m-p/198062#M160113 I know that below JSON path is wrong for product.csv and it should be like $.product.csv . Assume that before Fetchconfig , i will update the "filename1" attribute to product.csv or wholesale.csv or any filename in Updateattribute processor . Using that attribute , i need to get the value from the config file dynamically . I would need some work around to pass dynamic "filename" as attribute and extract the values for that attribute and store into next processor flowfile attribute or flowfile content . I am still new to Nifi expression language . Please assist . Currently i need to retrieve 1 attribute value passing dynamic attribute value like filename @MattWho @Shu_ashu @Nifi @NifiDev
... View more
Labels:
- Labels:
-
Apache NiFi
12-26-2020
09:40 PM
Hi All, I am building Nifi flow which fetch the filename.zip from shared server and unpack the .zip file then distributes the files across nifi cluster and finally put the files in HDFS. By doing this process , i am collecting success or failure status of each processors using custom message in Updateattribute processor . Refer the sample attached screen-print for updateattribute processor . Next i am planning to update overall status(success or error) of .zip file along with each file processed status ,like combining all information into one file and push the file to shared server to know about the status. I tried to use Mergecontent processor but i am unable to update the .zip file status and also noticed that if there is a delay in processing How to build the flowfile like below Filename.zip:staus:error_message Filename1.status:sucess_message Filename2:status:success_message Filename3:status:error_message Going forward i may also update the status message in JSON @Nifi @NifiDev
... View more
Labels:
- Labels:
-
Apache NiFi
12-24-2020
06:31 AM
wasimakram054 You can use ReplaceText processor to combine All the 15 attributes together and the result will be a single flowfile . Example in ReplaceText processor as below $attr1,$attr2,...$attr15 in replace text . You would need to link two flowfile queues to ReplaceText procssor and add the above example condition .
... View more
12-24-2020
03:34 AM
justenji I am able to build the success and error status with some custom messages using Updateattribute and converted those attributes using ReplaceText processor . I got some clue from the link you mentioned .Thank you
... View more
12-14-2020
03:17 PM
@justenji We can not use database in our platform to log the message from the flow .Can we use NFS fileystem to log the messages in a configuration file where it indicates success or failure with reason .if so could you please provide some nifi template . Thank you
... View more