Not so much resolved as worked around.  We had a large number of components on the canvas.  And we think we tracked the issue to timeouts in the cluster sync process.  (It was taking too long sometimes to get flow.xml sync'd between the nodes so it would stop.) The default timeout is 30 seconds.  We moved it up to 180 seconds. In Cloudera Manager, for the NiFi Configuration, search for properties which start with `nifi.cluster` to find the ones we changed. I also think things might have gotten better with more recent releases.  We have not seen this happen for quite some time. ... View more

It's really just the same message as the original.  (Un)fortunately, 2 of the 3 nodes of the cluster just dropped a few min ago so: 7/11/2024 16:57:12 UTC: Node disconnected due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. 07/11/2024 16:57:12 UTC: Disconnection requested due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. 07/11/2024 16:57:07 UTC: Node Status changed from CONNECTED to DISCONNECTED due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption. 07/11/2024 16:56:56 UTC: Received first heartbeat from connecting node. Node connected. 07/11/2024 16:56:54 UTC: Requesting that node connect to cluster ... View more

We are having this problem now with CFM 2.1.5.0 (NiFi 1.18) and have no later version at present we can upgrade to. It is not always the same node...but it may always be the primary node.  (At least it was the two times it happened today.) Are there particular usage patterns which you know that trigger this problem with node synchronization that we can try to avoid? Thanks. ... View more

It is with all data attempted.  It's a dynamic process, where the flowfile contents are mapped by an earlier QueryRecord processor to match the column names of the underlying table.  I have no access to the SQL, which is part of the problem, and this same process IS working against Postgres with no difference in the Avro file contents or destination table structure.  The postgres processor is configured the same with the exception of the Database type and DBCP controller service. And the error implies that the SQL is being generated incorrectly - feels like something is being left out somewhere.  Unfortunately, there does not seem to be a way to see the SQL NiFi is generating before sending to Oracle. I think the Oracle version on the other end is 19 - I know the JDBC driver is on the NiFi side.  But that also has not changed, and this same processor is still working on our 1.15 instance. (I went to check the AVRO reader config - not much to it.  It is just using the one embedded in the AVRO content.) And - for as much additional information as it provides (which isn't much), this is the complete bulletin.  I do not have access to see if there was additional logging. PutDatabaseRecord[id=8e509290-6652-385d-83e1-8e4788ae4086] Failed to put Records to database for FlowFile[filename=file_crmanalyticsam_useram_userpvo-batch792591341-20230907_230017.csv]. Routing to failure.: java.sql.BatchUpdateException: ORA-00936: missing expression ... View more

Yep!  Had moved on to that after your response and written up a JIRA for my team to build the needed tool.  Thanks again for the response.  My plan is the below   > Find all the processors which may have state, then check if each has state, and if so, clear it.   Find processors of used types which can contain state: e.g., QueryDatabaseTableRecord For each, pull the state by the ID and check if it has any stored state if totalEntryCount of cluster or local state is non-zero If so, call the clear state endpoint Query URL: /nifi-api/flow/search-results?q=QueryDatabaseTableRecord Pull Processor State: /nifi-api/processors/41584e33-adc6-171d-0000-0000581caccb/state Clear Processor State: POST /nifi-api/processors/41584e33-adc6-171d-0000-0000581caccb/state/clear-requests   ... View more

Thanks for the response.  TL;DR: It's complicated. 😄  I don't think my operations team will be a fan of shutting things down and purging files and/or zookeeper nodes.  But, this is good to know for when we have an important reason to purge state.   ... View more

Sure.  I had to make some extract it from a larger PG and make some adjustments due to key code being in a parameter.  But the remainder of the parameters are really placeholders for your environment information.  The main complication might be that I used a few external controller services that you will have to replace.  But their names in the externalControllerServices JSON should help you replace them.   ... View more

FWIW - I did set up a monitoring mechanism for bulletins.  I have not really made a way to automatically associate the bulletins back to the flowfiles, but I have utilized the cluster-level Controller Services for this.   Under Reporting Task Controller Services: DBCPConnectionPool for the database connection DatabaseRecordSink pointed at a table matching the internal BULLETINS data structure Under Reporting Tasks: QueryNiFiReportingTask which pulls from BULLETINS and structures into a record matching the table used by the above DatabaseRecordSink. The reporting task is set to run every 5 minutes, so that it captures each set of bulletins before it cycles out of the in-memory data structures. I then have other components reporting on records in that table for tracking by our other tools.  (And the flowfile ID is part of the bulletins structure - so it might be linkable back to whatever is stuck in the upstream connection.   ... View more

@apk132  I had something like this recently, but it was a result of a configuration error on the processor itself, and not anything to do with the flowfile.  Are you able to post the contents of the bulletin from the Bulletin Board screen? ... View more