First of all, your article is great! It was very useful on my first steps with NiFi and Kudu. Thanks!
My dataflow is:
I get data from REST API (for now it stopped, so you can see red square, but it works fine when i get data from it), then add needed attributes to dataflow (like timestamp, API's title), then split JSON array into JSON objests (no nested objects there, "key":"value" pairs only), then remove several fields via JOLT transformation (with symbols like '@'), then add dataflow attributes to JSON. For now i have flow contains JSON oblects with several "key":"value" data into each object. Keys like "field", "field1" or "field_name1". Values are numbers, strings(GUIDs or dates in different formats) or nulls. I checked JSON by stopping PutKudu and view data into queue.
My PutKudu configuration is:
I tried to use several Flush Mode values (AUTO_FLUSH_SYNC, AUTO_FLUSH_BACKGROUND), different Batch size values (2, 10, 100, 1000, 10000) and different Record Readers (AvroReader, several JsonPathReaders).
For now I extract 1 field from JSON object via JsonPathReader and store it into Kudu table with 1 column. JsonPathReader configuration is:
I checked that key "fieldid" exists in JSON and its value is GUID string. Table "impala::default.test_activities" with 1 string column "json_str" created via Impala. I added 1 test row to it. Also i checked existing of table with "kudu table list localhost:7051" command on kudu_master.
AvroSchemaRegistry configuration is:
and "All" schema is
{
"type": "record",
"name": "All",
"fields": [
{
"name" : "json_str",
"type" : "string"
}]
}Also i tried to use schema like this:
{
"type": "record",
"name": "All",
"fields": [
{
"name" : "json_str",
"type" : ["string","null"]
}]
}That what i have for now.
Thanks.
