Created 02-07-2024 02:27 AM
After the LookupRecord, we have the csv file of matched case(as shown in the table below). Then we want to UPDATE using PutDatabase Record for those that matched. The in between: UpdateRecord and JoltTransformations is to add in an attribute and select the attributes we want to use respectively. In the database, where the concat_columns are matched, we want to update that row in our database. However, the updating process takes very long. Could anyone provide some reasons to why?
On the other hand, there are unmatched scenarios, where we then INSERT the row to our database. That is successful and quick. Answers are greatly appreciated!
(our flow overview)
(UpdateRecord to add a column)
(jolt transformation details)
(Update PutdatabaseRecord Configurations)
(Dataset of matched)
REVISION_ID | FLAG | ID_BB_UNIQUE | FISCAL_YEAR_PERIOD | LATEST_PERIOD_END_DT_FULL_RECORD | ANNOUNCEMENT_DT | FUNDAMENTAL_ENTRY_DT | FUNDAMENTAL_UPDATE_DT | EQY_CONSOLIDATED | FILING_STATUS | ACCOUNTING_STANDARD | EQY_FUND_CRNCY | ID_BB_COMPANY | AVAILABLE_DATE | announcement_date | period_end | fiscal_period | fiscal_year | update_date | entry_date | Fundamentals_type_ID | COMPANY_ID | filing_status_ID | accounting_standard_ID | fiscal_period_ID | is_consolidated | Concat_columns | root | |||||||||||||||||||||||||
305945 | 1 | EQ0011769300001000 | 2024 Q2 | 20230930 | 20231024 | 20231114 | 20231120 | Y | OR | JP GAAP | 6 | 117693 | ######## | ######## | ######## | Q2 | 2024 | ######## | 11/14/2023 | 1 | 2964 | 2 | 67 | 22 | 1 | 2964_2024_2023-09-30_1_6_2_67_22_1_EQ0011769300001000 | MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=22, Filing_status_ID=2, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_2_67_22_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=356218684, CLEANUP_FLAG=1, Entry_date=2023-11-14}] | |||||||||||||||||||||||||
305945 | 1 | EQ0011769300001000 | 2024 Q2 | 20230930 | 20231024 | 20231114 | 20231120 | Y | MR | JP GAAP | 6 | 117693 | ######## | ######## | ######## | Q2 | 2024 | ######## | 11/14/2023 | 1 | 2964 | 1 | 67 | 22 | 1 | 2964_2024_2023-09-30_1_6_1_67_22_1_EQ0011769300001000 | MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=22, Filing_status_ID=1, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_1_67_22_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=355964254, CLEANUP_FLAG=1, Entry_date=2023-11-14}] | |||||||||||||||||||||||||
305945 | 1 | EQ0011769300001000 | 2024 C2 | 20230930 | 20231024 | 20231114 | 20231120 | Y | OR | JP GAAP | 6 | 117693 | ######## | ######## | ######## | C2 | 2024 | ######## | 11/14/2023 | 1 | 2964 | 2 | 67 | 32 | 1 | 2964_2024_2023-09-30_1_6_2_67_32_1_EQ0011769300001000 | MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=32, Filing_status_ID=2, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_2_67_32_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=356219117, CLEANUP_FLAG=1, Entry_date=2023-11-14}] | |||||||||||||||||||||||||
305945 | 1 | EQ0011769300001000 | 2024 C2 | 20230930 | 20231024 | 20231114 | 20231120 | Y | MR | JP GAAP | 6 | 117693 | ######## | ######## | ######## | C2 | 2024 | ######## | 11/14/2023 | 1 | 2964 | 1 | 67 | 32 | 1 | 2964_2024_2023-09-30_1_6_1_67_32_1_EQ0011769300001000 | MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=32, Filing_status_ID=1, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_1_67_32_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=355964310, CLEANUP_FLAG=1, Entry_date=2023-11-14}] | |||||||||||||||||||||||||
305945 | 1 | EQ0011769300001000 | 2024 S1 | 20230930 | 20231024 | 20231024 | 20231120 | Y | PR | JP GAAP | 6 | 117693 | ######## | ######## | ######## | S1 | 2024 | ######## | 10/24/2023 | 1 | 2964 | 3 | 67 | 11 | 1 | 2964_2024_2023-09-30_1_6_3_67_11_1_EQ0011769300001000 | MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=11, Filing_status_ID=3, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_3_67_11_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=355964207, CLEANUP_FLAG=1, Entry_date=2023-10-24}] |
Created 02-07-2024 11:33 AM
@ggyx Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @MattWho @steven-matison who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,