Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?

Solved Go to solution

What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?

Contributor

I have scenario where I am getting the file(stream:flowfile of NIFI) as of type csv file, then creating the dataframe and dumping it thats it. But after creating the dataframe the structure of the file got disturbed, if I open the same flowfile on my disk i could see clear structure with columns separated with tab, but with python dataframe I am not getting the same structure, if I get same structure i can perform row manipulation.

 

Here What I am doing:
1: using ExecuteSQL processor, I am getting database record,

2: then passing this record to ConvertRecord processor  to convert this avro record type csv file separated by tab.
convertRecordSetWriter settings...
Capture.PNG

The output of the flowfile is ...

Capture.PNG


3: Then the reading flowfile(step 2 folowfile) as python data using ExecuteStreamCommand, coz I am want to perform some action on the database record, to do this my record structure has been changed in data frame.
Capture.PNG

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?

Contributor

it did work after adding '\t' to read_csv as 2nd arg.

View solution in original post

1 REPLY 1
Highlighted

Re: What is wrong in reading Nifi flowfile and creating python(pandas)dataframe?

Contributor

it did work after adding '\t' to read_csv as 2nd arg.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here