Member since
03-17-2018
5
Posts
0
Kudos Received
0
Solutions
03-18-2018
10:36 AM
Thanks @Rahul Soni, the JoltTransformJson processor works for me. Also thanks to @Shu for explaining everything.
... View more
03-18-2018
03:12 AM
Hi @Shu, Thanks for your prompt response. Your reply has given me new insights on how to handle my problems. The data which I provided initially was very simplified, as I assume that it was just a regex question. Here's a better representation of the data. There's around 50 fields for each record and around 1,500 records in each json file. {
"XMLFile_2234.DAILY": {
"xsi:schemaLocation": "http://xml.mscibarra.com/random.xsd",
"dataset_12232": {
"entry": [
{
"record_date": "2017-03-01",
"code": "233432",
"country": "USA",
"inter_com_value": ".STRATE",
"country_code": "US",
"One_code": "1",
"Two_code": "0",
"Three_code": "1",
"value_code": "0",
"big_code": "1",
"small_code": "0",
"mid_code": "0",
"exist_code": "1",
"restricted_code": "0",
"base_flag": "0",
"emply_count": "225",
"unadj_reference_value": "5465.546456",
"ref_date": "2016-05-31",
"old_date": "2013-05-31",
"new_date": "2014-05-31",
"value_type": "EMTE",
"estval_old": "2321.123543",
"estval_new": "2354.585674",
"world_code_type": "MTEE",
"world_code_old": "1232.163564",
"world_code_new": "1432.67565",
"region_code_type": "TMRQ",
"region_code_old": "2343.476576",
"region_code_new": "6546.678576",
"mkt_based_adj": "76856.325425",
"total_sale_value_weighted": "23423.565434",
"total_sale_value_raw": "23423.453535",
"normalised_value_one": "1000.000000",
"normalised_value_two": "1000.000000",
"normalised_value_three": "1000.000000",
"moving_value_one": "98456754.363246300000000",
"moving_value_one_nd": "98456754.363246300000000",
"moving_value_two": "98456754.363246300000000",
"moving_value_two_nd": "98456754.363246300000000",
"moving_value_three": "98456754.363246300000000",
"moving_value_three_nd": "98456754.363246300000000",
"moving_indice_pt_one": "0.000000000000000",
"moving_indice_pt_one_p": "0.46789870755657",
"moving_indice_pt_two": "0.000000000000000",
"moving_indice_pt_two_p": "0.46789870755657",
"moving_indice_pt_three": "0.000000000000000",
"moving_indice_pt_three_p": "0.46789870755657",
"moving_indice_pt_four": "0.000000000000000",
"moving_indice_pt_four_p": "0.46789870755657"
},
{
"record_date": "2017-03-01",
"code": "236453",
"country": "VEN",
"inter_com_value": ".STRATE",
"country_code": "VE",
"One_code": "1",
"Two_code": "0",
"Three_code": "1",
"value_code": "0",
"big_code": "1",
"small_code": "0",
"mid_code": "0",
"exist_code": "1",
"restricted_code": "0",
"base_flag": "0",
"emply_count": "244",
"unadj_reference_value": "5465.546456",
"ref_date": "2016-05-31",
"old_date": "2013-05-31",
"new_date": "2014-05-31",
"value_type": "EMTE",
"estval_old": "2321.123543",
"estval_new": "2354.585674",
"world_code_type": "MTEE",
"world_code_old": "1232.163564",
"world_code_new": "1432.67565",
"region_code_type": "TMRQ",
"region_code_old": "2343.476576",
"region_code_new": "6546.678576",
"mkt_based_adj": "76856.325425",
"total_sale_value_weighted": "23423.565434",
"total_sale_value_raw": "23423.453535",
"normalised_value_one": "1000.000000",
"normalised_value_two": "1000.000000",
"normalised_value_three": "1000.000000",
"moving_value_one": "98456754.363246300000000",
"moving_value_one_nd": "98456754.363246300000000",
"moving_value_two": "98456754.363246300000000",
"moving_value_two_nd": "98456754.363246300000000",
"moving_value_three": "98456754.363246300000000",
"moving_value_three_nd": "98456754.363246300000000",
"moving_indice_pt_one": "0.000000000000000",
"moving_indice_pt_one_p": "0.46789870755657",
"moving_indice_pt_two": "0.000000000000000",
"moving_indice_pt_two_p": "0.46789870755657",
"moving_indice_pt_three": "0.000000000000000",
"moving_indice_pt_three_p": "0.46789870755657",
"moving_indice_pt_four": "0.000000000000000",
"moving_indice_pt_four_p": "0.46789870755657"
}
]
}
}
} I've tried both your suggested methods. For Method 1, I can't construct a Json Expression that allows me to include the entire string of "XMLFile_2234.DAILY" as part of the expression mainly due to the period in the string. Also, if the header value changes from file to file, can I assume that this method may not be suitable? For Method 2, the ExtractText processor does not seem to be able to extract any value by using the following. "entry":(.*]) Instead I tried the following expression. \[([^]]+)\] And I got the following values in the attributes. The values seems to be truncated and the 2nd record is not picked up. Also, if I have around 1,500 records within each Json file that need to be split, will this method of using attributes have any limitations? Thanks
... View more
03-17-2018
06:55 AM
I have a json file and would like to just keep everything within [ ] so that i can send the flowfile into SplitJson processor and subsequently into elasticsearch. A sample of the content of the file is attached below. {
"XMLfile_2234": {
"xsi:schemaLocation": "http://xml.mscibarra.com/random.xsd",
"dataset_12232": {
"entry": [
{
"record_date": "2017-03-01",
"country": "USA",
"funds": "100",
},
{
"record_date": "2018-03-01",
"country": "Chile",
"funds": "10000",
}
]
}
}
} How do i remove all text and character before and after the [ ]? I would want to keep the square brackets too. I'm a complete noob with regards to regex and replacetext. Thanks for the help.
... View more