Skip to main content
ipson
ipson
Gold ⭐️
October 23, 2023
Solved

Stateful processors in test flows vs. deployment

In Cloudera Data Flow, I'm using the `QueryDatabaseTable` processor with a maximum-value column to keep track of which records have already been processed. When I'm in Flow Designer testing the flow, I can see that it's maintaining state. Does this state persist if I deploy it, or does the deploy clear the state? I expect it would be the latter, but want to check.

Follow up question: Say I make changes to the flow and redeploy. Is the state cleared this time, or does it use the last deployments flow? I'd like to be confident that making changes to the flow doesn't cause it to reprocess all the data again.

    DigitalPlumber
    DigitalPlumberAnswer
    Gold ⭐️
    October 30, 2023

    Hello @ipson first sorry my previous comment was not posted for some reason.

    It is great to see you are exploring DataFlow Data Service on Cloudera Public Cloud.
    To answer your question:

    Does it persist state?

    Unfortunately there is currently no ability to change the flow on a running deployment, so you'd have to
    • publish v2 to the catalog from flow designer
    • terminate original deployment
    • perform new deployment with v2


    If you have to deploy a new version of the flow it means you are standing up a new namespace and in turn  it will have a new zookeeper / NiFi / disk among other things contained within the kubernetes namespace.

    View in thread

    2 replies

    DigitalPlumber
    Gold ⭐️
    DigitalPlumberAnswer
    Gold ⭐️
    October 30, 2023

    Hello @ipson first sorry my previous comment was not posted for some reason.

    It is great to see you are exploring DataFlow Data Service on Cloudera Public Cloud.
    To answer your question:

    Does it persist state?

    Unfortunately there is currently no ability to change the flow on a running deployment, so you'd have to
    • publish v2 to the catalog from flow designer
    • terminate original deployment
    • perform new deployment with v2


    If you have to deploy a new version of the flow it means you are standing up a new namespace and in turn  it will have a new zookeeper / NiFi / disk among other things contained within the kubernetes namespace.

    Translate
      DianaTorres
      Community Manager
      DianaTorres
      Community Manager
      November 2, 2023

      @ipson Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.

      Translate
      Regards, Diana Torres, Senior Community Moderator
      Terms & ConditionsAccessibility statement