Support Questions

Find answers, ask questions, and share your expertise

NiFi Record validation and putKudu

avatar
Contributor

Hello,

 

This may be a silly question but I'm in a bit of a loss and have been stuck for a couple days now trying to find a solution.

 

I have a flow design where I have to validate a record against an avro schema and insert that record in a Kudu table.

 

The avro validation schema is:

 

 

{ 
	"type" : "record", 
	"name" : "tbcertificati", 
	"fields" : [  
				{ "name" : "num_caso", "type" : "string" },
				{ "name" : "num_rch", "type" : "string" },
				{ "name" : "num_cert", "type" : "string" },
				{ "name" : "seqno", "type" : "string" },
				{ "name" : "data_inizio_kudu", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "num_vis", "type" : "double" },
				{ "name" : "data_compilaz_cert", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "data_acquisiz_cert", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "cod_matr_med", "type" : "double" },
				{ "name" : "medico_struttura", "type" : "string" },
				{ "name" : "flag_int_ext", "type" : "string" },
				{ "name" : "cod_cert", "type" : "string" },
				{ "name" : "cod_prognosi", "type" : "double" },
				{ "name" : "data_iniz_prognosi", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "data_fine_prognosi", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "desc_cert", "type" : "string" },
				{ "name" : "data_ripresa_lav", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "flag_postumi", "type" : "string" },
				{ "name" : "flag_medico_stru", "type" : "string" },
				{ "name" : "note", "type" : "string" },
				{ "name" : "prg", "type" : "double" },
				{ "name" : "crt_time", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "id_usr_crt", "type" : "string" },
				{ "name" : "mod_time", "type" : { 
								"type":"long", 
								"logicalType":"timestamp-millis"}},
				{ "name" : "id_usr_mod", "type" : "string" },
				{ "name" : "fl_annullato", "type" : "string" },
				{ "name" : "dm_timestamp", "type" : "string" },
				{ "name" : "dm_txid", "type" : "string" },
				{ "name" : "dm_operation_type", "type" : "string" }
]
}

 

 

 

A record can arrive like this:

 

 

{
  "DM_TIMESTAMP" : "1970-01-01 00:00:00.000",
  "DM_TXID" : "1",
  "DM_OPERATION_TYPE" : "A",
  "DM_USER" : "STRING",
  "NUM_CASO" : "1",
  "NUM_RCH" : "1",
  "NUM_CERT" : "1",
  "NUM_VIS" : "",
  "DATA_COMPILAZ_CERT" : "1970-01-01 00:00:00",
  "DATA_ACQUISIZ_CERT" : "",
  "COD_MATR_MED" : "",
  "MEDICO_STRUTTURA" : "",
  "FLAG_INT_EXT" : "A",
  "COD_CERT" : "A",
  "COD_PROGNOSI" : "1",
  "DATA_INIZ_PROGNOSI" : "",
  "DATA_FINE_PROGNOSI" : "",
  "DESC_CERT" : "",
  "DATA_RIPRESA_LAV" : "1970-01-01",
  "FLAG_POSTUMI" : "",
  "FLAG_MEDICO_STRU" : "A",
  "NOTE" : "",
  "PRG" : "1",
  "CRT_TIME" : "1970-01-01 00:00:00",
  "ID_USR_CRT" : "STRING",
  "MOD_TIME" : "1970-01-01 00:00:00",
  "ID_USR_MOD" : "STRING",
  "FL_ANNULLATO" : "A",
  "SEQNO" : "1",
  "DATA_INIZIO_KUDU" : 1673274352029
}

 

 

 

The putKudu processor is as following:

linssab_0-1673514294007.png

Validation record always fails, and if trying to directly insert the record with the PutKudu processor, I get the error:

 

 

Failed to write due to Can't set primary key column foo to null : java.land.IllegalArgumentException: Can't set primary key column num_caso to null

 

 

 

I have trying capitalizing/decapitalizing the field names, with no success. The field clearly has a value associated to it, I don't understand why it is said to be null.

 

Thanks in advance.

0 REPLIES 0