Support Questions

Find answers, ask questions, and share your expertise

PutDatabaseRecord UPDATE takes too long

avatar
New Contributor

After the LookupRecord, we have the csv file of matched case(as shown in the table below). Then we want to UPDATE using PutDatabase Record for those that matched. The in between: UpdateRecord and JoltTransformations is to add in an attribute and select the attributes we want to use respectively. In the database, where the concat_columns are matched, we want to update that row in our database. However, the updating process takes very long. Could anyone provide some reasons to why?

On the other hand,  there are unmatched scenarios, where we then INSERT the row to our database. That is successful and quick. Answers are greatly appreciated! 

(our flow overview)

ggyx_1-1707300878884.png

(UpdateRecord to add a column) 

ggyx_3-1707301613988.png

(jolt transformation details)

ggyx_2-1707301206078.png

(Update PutdatabaseRecord Configurations)

ggyx_0-1707300822172.png

(Dataset of matched) 

REVISION_IDFLAGID_BB_UNIQUEFISCAL_YEAR_PERIODLATEST_PERIOD_END_DT_FULL_RECORDANNOUNCEMENT_DTFUNDAMENTAL_ENTRY_DTFUNDAMENTAL_UPDATE_DTEQY_CONSOLIDATEDFILING_STATUSACCOUNTING_STANDARDEQY_FUND_CRNCYID_BB_COMPANYAVAILABLE_DATEannouncement_dateperiod_endfiscal_periodfiscal_yearupdate_dateentry_dateFundamentals_type_IDCOMPANY_IDfiling_status_IDaccounting_standard_IDfiscal_period_IDis_consolidatedConcat_columnsroot                         
3059451EQ00117693000010002024 Q220230930202310242023111420231120YORJP GAAP6117693########################Q22024########11/14/2023129642672212964_2024_2023-09-30_1_6_2_67_22_1_EQ0011769300001000MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=22, Filing_status_ID=2, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_2_67_22_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=356218684, CLEANUP_FLAG=1, Entry_date=2023-11-14}]
3059451EQ00117693000010002024 Q220230930202310242023111420231120YMRJP GAAP6117693########################Q22024########11/14/2023129641672212964_2024_2023-09-30_1_6_1_67_22_1_EQ0011769300001000MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=22, Filing_status_ID=1, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_1_67_22_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=355964254, CLEANUP_FLAG=1, Entry_date=2023-11-14}]
3059451EQ00117693000010002024 C220230930202310242023111420231120YORJP GAAP6117693########################C22024########11/14/2023129642673212964_2024_2023-09-30_1_6_2_67_32_1_EQ0011769300001000MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=32, Filing_status_ID=2, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_2_67_32_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=356219117, CLEANUP_FLAG=1, Entry_date=2023-11-14}]
3059451EQ00117693000010002024 C220230930202310242023111420231120YMRJP GAAP6117693########################C22024########11/14/2023129641673212964_2024_2023-09-30_1_6_1_67_32_1_EQ0011769300001000MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=32, Filing_status_ID=1, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_1_67_32_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=355964310, CLEANUP_FLAG=1, Entry_date=2023-11-14}]
3059451EQ00117693000010002024 S120230930202310242023102420231120YPRJP GAAP6117693########################S12024########10/24/2023129643671112964_2024_2023-09-30_1_6_3_67_11_1_EQ0011769300001000MapRecord[{Announcement_date=2023-10-24, ID_BB_COMPANY=117693, Fiscal_period_ID=11, Filing_status_ID=3, ID_BB_UNIQUE=EQ0011769300001000, Is_consolidated=1, Fiscal_year=2024, EQY_FUND_CRNCY=6, Flag=1, Update_date=2024-01-04, Fundamentals_type_ID=1, Concat_columns=2964_2024_2023-09-30_1_6_3_67_11_1_EQ0011769300001000, Accounting_standard_ID=67, Latest_period_end=2023-09-30, U3_ID_FS=null, Revision=305442, Company_ID=2964, Available_date=2023-10-24, Update_lock=0, ID=355964207, CLEANUP_FLAG=1, Entry_date=2023-10-24}]
1 REPLY 1

avatar
Community Manager

@ggyx Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @steven-matison  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.


Regards,

Diana Torres,
Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community: