What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?

murali2425 — Tue, 28 Apr 2020 14:47:37 GMT

I have scenario where I am getting the file(stream:flowfile of NIFI) as of type csv file, then creating the dataframe and dumping it thats it. But after creating the dataframe the structure of the file got disturbed, if I open the same flowfile on my disk i could see clear structure with columns separated with tab, but with python dataframe I am not getting the same structure, if I get same structure i can perform row manipulation.

Here What I am doing:
1: using ExecuteSQL processor, I am getting database record,

2: then passing this record to ConvertRecord processor  to convert this avro record type csv file separated by tab.
convertRecordSetWriter settings...



The output of the flowfile is ...




3: Then the reading flowfile(step 2 folowfile)  as python data using ExecuteStreamCommand, coz I am want to perform some action on the database record, to do this my record structure has been changed  in data frame.

Re: What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?

murali2425 — Tue, 28 Apr 2020 18:02:29 GMT

it did work after adding '\t' to read_csv as 2nd arg.

question What is wrong in reading Nifi flowfile and creating python(pandas)dataframe? in Support Questions

What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?

Re: What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?