Nifi set Databases processor state in zookeeper

[#1]

Hi All,

I use 2 standard processors GenerateTableFetch and QueryDatabaseTable.

For some reason I need to set state for GenerateTableFetch processor from QueryDatabaseTable. I checked previous value in zookeeper and it very strange, with invisible characters. I made copy-paste from one to another, without success.

Any ideas how I can change state for this processors manualy?

Thanks

[#2 · Accepted solution]

Hello @Ilya Li

NiFi stores states as binary representation starting with encoding version. That's why you see invisible characters I believe, and copy-paste will not work [1].

Theoretically, but zk-migrator.sh would be able to copy zookeeper state which is available in NiFi Toolkit [2]. I did a simple experimentation using zk-migrator.sh. I was able to copy state from a processor to another by following steps:

1. Export state from Zk to a JSON file:

./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk.json -r

2. Copy exported zk.json to a different file, and edit:

Changed connectString and servers to 'localhost:2181' as the tool doesn't allow loading data to the same destination. Also removed other paths. So that the file only contains two objects, one containing 'connectString' and the other containing exported state that you want to copy. Change the path with target processor UUID, so that it will be loaded into the target processor:

[
  {
    "connectString": "localhost:2182",
    "servers": [
      "localhost:2182"
    ],
    "path": "/"
  },
  {
    "path": "/nifi/components/05f810b2-015d-1000-ffff-ffff80b2e488",
    "data": [1, 0, ...],    "stat": {
      "czxid": 552,
      "mzxid": 553,
      "ctime": 1499042915091,
      "mtime": 1499042915154,
      "version": 1,
      "cversion": 0,
      "aversion": 0,
      "ephemeralOwner": 0,
      "dataLength": 79,
      "numChildren": 0,
      "pzxid": 552
    },
    "acls": [
      {
        "perms": 31,
        "id": {
          "scheme": "world",
          "id": "anyone"
        }
      }
    ],
    "ephemeralOwner": 0
  }
]

3. Load modified zk json:

./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk-copy.dat -s

This way, you can copy state from a processor to another. It may work if those processors use the same structure of state (same keys ... etc).

Alternative way is to write a simple Zk client app referring ZookeeperStateProvider's serialize and deserialize methods [1].

[1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nif...

[2] https://nifi.apache.org/download.html

Hope this helps,

Koji

[#3]

Thanks Koji,

This was very helpful.

May be do you know any ZK client app which use ZookeeperStateProvider's serialize and deserialize methods?

[#4]

Unfortunately, I'm not aware of any existing client app that can deserialize what NiFi state manager stores. But since ZookeeperStateProvider.deserialize method source code is available and it's not that complicated, you can write a simple app that connects to Zk, get value from Znode and deserialize it.