Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi Record validation and putKudu

avatar
Contributor

Hello,

 

This may be a silly question but I'm in a bit of a loss and have been stuck for a couple days now trying to find a solution.

 

I have a flow design where I have to validate a record against an avro schema and insert that record in a Kudu table.

 

The avro validation schema is:

 

 

{ 
	"type" : "record", 
	"name" : "tbcertificati", 
	"fields" : [  
				{ "name" : "num_caso", "type" : "string" },
				{ "name" : "num_rch", "type" : "string" },
				{ "name" : "num_cert", "type" : "string" },
				{ "name" : "seqno", "type" : "string" },
				{ "name" : "data_inizio_kudu", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "num_vis", "type" : "double" },
				{ "name" : "data_compilaz_cert", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "data_acquisiz_cert", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "cod_matr_med", "type" : "double" },
				{ "name" : "medico_struttura", "type" : "string" },
				{ "name" : "flag_int_ext", "type" : "string" },
				{ "name" : "cod_cert", "type" : "string" },
				{ "name" : "cod_prognosi", "type" : "double" },
				{ "name" : "data_iniz_prognosi", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "data_fine_prognosi", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "desc_cert", "type" : "string" },
				{ "name" : "data_ripresa_lav", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "flag_postumi", "type" : "string" },
				{ "name" : "flag_medico_stru", "type" : "string" },
				{ "name" : "note", "type" : "string" },
				{ "name" : "prg", "type" : "double" },
				{ "name" : "crt_time", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "id_usr_crt", "type" : "string" },
				{ "name" : "mod_time", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "id_usr_mod", "type" : "string" },
				{ "name" : "fl_annullato", "type" : "string" },
				{ "name" : "dm_timestamp", "type" : "string" },
				{ "name" : "dm_txid", "type" : "string" },
				{ "name" : "dm_operation_type", "type" : "string" }
]
}

 

 

 

A record can arrive like this:

 

 

{
  "DM_TIMESTAMP" : "1970-01-01 00:00:00.000",
  "DM_TXID" : "1",
  "DM_OPERATION_TYPE" : "A",
  "DM_USER" : "STRING",
  "NUM_CASO" : "1",
  "NUM_RCH" : "1",
  "NUM_CERT" : "1",
  "NUM_VIS" : "",
  "DATA_COMPILAZ_CERT" : "1970-01-01 00:00:00",
  "DATA_ACQUISIZ_CERT" : "",
  "COD_MATR_MED" : "",
  "MEDICO_STRUTTURA" : "",
  "FLAG_INT_EXT" : "A",
  "COD_CERT" : "A",
  "COD_PROGNOSI" : "1",
  "DATA_INIZ_PROGNOSI" : "",
  "DATA_FINE_PROGNOSI" : "",
  "DESC_CERT" : "",
  "DATA_RIPRESA_LAV" : "1970-01-01",
  "FLAG_POSTUMI" : "",
  "FLAG_MEDICO_STRU" : "A",
  "NOTE" : "",
  "PRG" : "1",
  "CRT_TIME" : "1970-01-01 00:00:00",
  "ID_USR_CRT" : "STRING",
  "MOD_TIME" : "1970-01-01 00:00:00",
  "ID_USR_MOD" : "STRING",
  "FL_ANNULLATO" : "A",
  "SEQNO" : "1",
  "DATA_INIZIO_KUDU" : 1673274352029
}

 

 

 

The putKudu processor is as following:

linssab_0-1673514294007.png

Validation record always fails, and if trying to directly insert the record with the PutKudu processor, I get the error:

 

 

Failed to write due to Can't set primary key column foo to null : java.land.IllegalArgumentException: Can't set primary key column num_caso to null

 

 

 

I have trying capitalizing/decapitalizing the field names, with no success. The field clearly has a value associated to it, I don't understand why it is said to be null.

 

Thanks in advance.

0 REPLIES 0