Member since
11-22-2017
2
Posts
1
Kudos Received
0
Solutions
11-23-2017
08:37 AM
First of all, your article is great! It was very useful on my first steps with NiFi and Kudu. Thanks! My dataflow is: I get data from REST API (for now it stopped, so you can see red square, but it works fine when i get data from it), then add needed attributes to dataflow (like timestamp, API's title), then split JSON array into JSON objests (no nested objects there, "key":"value" pairs only), then remove several fields via JOLT transformation (with symbols like '@'), then add dataflow attributes to JSON. For now i have flow contains JSON oblects with several "key":"value" data into each object. Keys like "field", "field1" or "field_name1". Values are numbers, strings(GUIDs or dates in different formats) or nulls. I checked JSON by stopping PutKudu and view data into queue. My PutKudu configuration is: I tried to use several Flush Mode values (AUTO_FLUSH_SYNC, AUTO_FLUSH_BACKGROUND), different Batch size values (2, 10, 100, 1000, 10000) and different Record Readers (AvroReader, several JsonPathReaders). For now I extract 1 field from JSON object via JsonPathReader and store it into Kudu table with 1 column. JsonPathReader configuration is: I checked that key "fieldid" exists in JSON and its value is GUID string. Table "impala::default.test_activities" with 1 string column "json_str" created via Impala. I added 1 test row to it. Also i checked existing of table with "kudu table list localhost:7051" command on kudu_master. AvroSchemaRegistry configuration is: and "All" schema is {
"type": "record",
"name": "All",
"fields": [
{
"name" : "json_str",
"type" : "string"
}]
} Also i tried to use schema like this: {
"type": "record",
"name": "All",
"fields": [
{
"name" : "json_str",
"type" : ["string","null"]
}]
} That what i have for now. Thanks.
... View more
11-22-2017
02:23 PM
1 Kudo
I'm trying to write data to Kudu using PutKudu processor. PutKudu processor's log looks like this: 2017-11-22 15:17:02,416 INFO [NiFi Web Server-2783] o.a.n.c.s.StandardProcessScheduler Starting PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515]
2017-11-22 15:17:02,420 DEBUG [StandardProcessScheduler Thread-2] org.apache.nifi.processors.kudu.PutKudu PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] Setting up Kudu connection...
2017-11-22 15:17:02,663 DEBUG [StandardProcessScheduler Thread-2] org.apache.nifi.processors.kudu.PutKudu PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] Kudu connection successfully initialized
2017-11-22 15:17:02,664 INFO [StandardProcessScheduler Thread-2] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] to run with 1 threads
2017-11-22 15:17:02,670 INFO [Timer-Driven Process Thread-10] org.apache.nifi.processors.kudu.PutKudu PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] KUDU: number of inserted records: 1
2017-11-22 15:17:02,674 INFO [Timer-Driven Process Thread-7] org.apache.nifi.processors.kudu.PutKudu PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] KUDU: number of inserted records: 1
2017-11-22 15:17:02,676 INFO [Timer-Driven Process Thread-7] org.apache.nifi.processors.kudu.PutKudu PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] KUDU: number of inserted records: 1
2017-11-22 15:17:02,678 INFO [Timer-Driven Process Thread-7] org.apache.nifi.processors.kudu.PutKudu PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] KUDU: number of inserted records: 1
2017-11-22 15:17:02,681 INFO [Timer-Driven Process Thread-7] org.apache.nifi.processors.kudu.PutKudu PutKudu[id=ca52eca0-015f-1000-c66f-d789248d6515] KUDU: number of inserted records: 1 But no data in Kudu and no errors in Kudu logs. Kudu is working great via Impala. Any ideas? Thanks
... View more
Labels:
- Labels:
-
Apache Kudu
-
Apache NiFi