Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi set Databases processor state in zookeeper

Solved Go to solution
Highlighted

Nifi set Databases processor state in zookeeper

Contributor

Hi All,

I use 2 standard processors GenerateTableFetch and QueryDatabaseTable.

For some reason I need to set state for GenerateTableFetch processor from QueryDatabaseTable. I checked previous value in zookeeper and it very strange, with invisible characters. I made copy-paste from one to another, without success.

Any ideas how I can change state for this processors manualy?

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi set Databases processor state in zookeeper

Hello @Ilya Li

NiFi stores states as binary representation starting with encoding version. That's why you see invisible characters I believe, and copy-paste will not work [1].

Theoretically, but zk-migrator.sh would be able to copy zookeeper state which is available in NiFi Toolkit [2]. I did a simple experimentation using zk-migrator.sh. I was able to copy state from a processor to another by following steps:

1. Export state from Zk to a JSON file:

./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk.json -r

2. Copy exported zk.json to a different file, and edit:

Changed connectString and servers to 'localhost:2181' as the tool doesn't allow loading data to the same destination. Also removed other paths. So that the file only contains two objects, one containing 'connectString' and the other containing exported state that you want to copy. Change the path with target processor UUID, so that it will be loaded into the target processor:

[
  {
    "connectString": "localhost:2182",
    "servers": [
      "localhost:2182"
    ],
    "path": "/"
  },
  {
    "path": "/nifi/components/05f810b2-015d-1000-ffff-ffff80b2e488",
    "data": [1, 0, ...],    "stat": {
      "czxid": 552,
      "mzxid": 553,
      "ctime": 1499042915091,
      "mtime": 1499042915154,
      "version": 1,
      "cversion": 0,
      "aversion": 0,
      "ephemeralOwner": 0,
      "dataLength": 79,
      "numChildren": 0,
      "pzxid": 552
    },
    "acls": [
      {
        "perms": 31,
        "id": {
          "scheme": "world",
          "id": "anyone"
        }
      }
    ],
    "ephemeralOwner": 0
  }
]

3. Load modified zk json:

./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk-copy.dat -s

This way, you can copy state from a processor to another. It may work if those processors use the same structure of state (same keys ... etc).

Alternative way is to write a simple Zk client app referring ZookeeperStateProvider's serialize and deserialize methods [1].

[1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nif...

[2] https://nifi.apache.org/download.html

Hope this helps,

Koji

3 REPLIES 3

Re: Nifi set Databases processor state in zookeeper

Hello @Ilya Li

NiFi stores states as binary representation starting with encoding version. That's why you see invisible characters I believe, and copy-paste will not work [1].

Theoretically, but zk-migrator.sh would be able to copy zookeeper state which is available in NiFi Toolkit [2]. I did a simple experimentation using zk-migrator.sh. I was able to copy state from a processor to another by following steps:

1. Export state from Zk to a JSON file:

./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk.json -r

2. Copy exported zk.json to a different file, and edit:

Changed connectString and servers to 'localhost:2181' as the tool doesn't allow loading data to the same destination. Also removed other paths. So that the file only contains two objects, one containing 'connectString' and the other containing exported state that you want to copy. Change the path with target processor UUID, so that it will be loaded into the target processor:

[
  {
    "connectString": "localhost:2182",
    "servers": [
      "localhost:2182"
    ],
    "path": "/"
  },
  {
    "path": "/nifi/components/05f810b2-015d-1000-ffff-ffff80b2e488",
    "data": [1, 0, ...],    "stat": {
      "czxid": 552,
      "mzxid": 553,
      "ctime": 1499042915091,
      "mtime": 1499042915154,
      "version": 1,
      "cversion": 0,
      "aversion": 0,
      "ephemeralOwner": 0,
      "dataLength": 79,
      "numChildren": 0,
      "pzxid": 552
    },
    "acls": [
      {
        "perms": 31,
        "id": {
          "scheme": "world",
          "id": "anyone"
        }
      }
    ],
    "ephemeralOwner": 0
  }
]

3. Load modified zk json:

./bin/zk-migrator.sh -z localhost:2181 -f /tmp/zk-copy.dat -s

This way, you can copy state from a processor to another. It may work if those processors use the same structure of state (same keys ... etc).

Alternative way is to write a simple Zk client app referring ZookeeperStateProvider's serialize and deserialize methods [1].

[1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nif...

[2] https://nifi.apache.org/download.html

Hope this helps,

Koji

Re: Nifi set Databases processor state in zookeeper

Contributor

Thanks Koji,

This was very helpful.

May be do you know any ZK client app which use ZookeeperStateProvider's serialize and deserialize methods?

Re: Nifi set Databases processor state in zookeeper

Unfortunately, I'm not aware of any existing client app that can deserialize what NiFi state manager stores. But since ZookeeperStateProvider.deserialize method source code is available and it's not that complicated, you can write a simple app that connects to Zk, get value from Znode and deserialize it.