CSVReader and CSVRecordSetWriter doesn't consider interger values present in double quotes as string

Hi @phadkev ,

I'm not sure if there is a way around this without using Schema. You mentioned that you want to read the CSV files dynamically but does that mean you dont know what kind of CSV data you will get ? if so how are you parsing the data after conversion? what processor are you using the CSVReader & Write in ? maybe you can elaborate more on what you are trying to do with the data beginning to end to see if we can help in another way.

Hi @SAMSAL ,

I'm using the CSV readers and writers in Query Record processor and CSVtoJSON Record Converter processor. 

I am expecting around 10 types of CSV files(I mean to say, files with different columns). But this number might increase in the future upto 50. Even though I know what will these 50 files exactly contain, I'll still have to write all the 50 schemas in a schema registry. Which is complex to maintain and work with. 

So I thought of this method in which I'll append all the values in a csv file with double quotes such that the reader and writer will consider it as a string. But it still continues to read it as an integer

Input File:

col1,col2,col3,col4,col5
999,C10,100,010,0
999,C06,10,010,0

This is what the CSV writer is giving:

Hi @phadkev ,

Since you dont care if all the values come as string , then I would suggest doing the following:

1- Use the ExtractRecordSchema processor to generate the avro.schema attribute with the record schema as inferred by Nifi (Available 1.26+ version ) as follows:

The CSVReader for this processor will use infer schema strategy.

once you pass your CSV input through this processor you will have new flowfile attribute avro.schema with the following value

As Expected some of the values like name are assigned int type which we will take care of in the next step.

2- Use UpdateAttribute to replace any int type with string type inside the avro.schema attribute as follows:

The Expression Language used to re set the avro.schme attribute:

${avro.schema:replace("int","string")}

3- Use the QueryRecord with a different CSVReader from step 1  where this one uses the "use Schema Text Property" . Notice how by default the Shema Text property is set to the avro.schema attribute, which we generated from step 1&2:

Also make sure you set the same strategy for the CSVRecordWriter to ensure that the read and written CSV will be in  the desired format.

Hope that helps. If it does, please accept the solution.

Thanks

Hi @SAMSAL ,

Thanks this did work.