Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

can you serialize data with NiFi?

Highlighted

can you serialize data with NiFi?

Expert Contributor

Hi, I am trying to send a list to a specific port with NiFi. However, i am getting issues when the server receives the list. I think an option is to serialize the list before sending it. Is this possible with Nifi or should i use my own code with ExecuteStreamCommand processor? Thank you

6 REPLIES 6
Highlighted

Re: can you serialize data with NiFi?

New Contributor

@Roger Young ,

Which processors are you using? Sending data to a port probably involves using the HandleHTTPRequest processor.

How is the "list" being generated? Is it coming from a file, HTTPRequest, email, or something else?

Highlighted

Re: can you serialize data with NiFi?

Expert Contributor

Hi @Nathan Maynes. The list is coming from a file. I am sending it to a python server on the local machine via a PutTcp processor. But every now and then the python server gives an error.

Highlighted

Re: can you serialize data with NiFi?

New Contributor

Can you describe your intended flow in more detail? Choosing the PutTcp processor to send files is alright for low volume workloads so I would guess you are attempting to have a flow move files from one machine to another, process the file by "record"(I use the term in a very general sense to mean any data that are meant to be kept together), then do something else with the individual records. That sounds much easier with a GetFile, SplitFile, UpdateAttribute type of workflow.

Highlighted

Re: can you serialize data with NiFi?

Expert Contributor

Hi @Nathan Maynes

Im using GetFile and SplitFile to take the rows of data from the file (which gives me a flowfile consisting of 30 features) then set a control rate of 1 flowfile per second before sending it to PutTCP processor. I am trying to emulate a device sending data in real time to my server on localhost, where i perform processing on the incoming flowfile. But the server gives an error which seems to be related to the format of the incoming flowfile.

Highlighted

Re: can you serialize data with NiFi?

New Contributor

This sounds like a fun troubleshooting session~!

First, I would double check you are sending the correct delimiter characters and that the file body does not contain them. You can do this by routing the failure relationship to an inactive processor. Examine the queue list and check out some of the FlowFiles in data provenance. Try checking the logs in {path-to}/{nifi}/logs/nifi-app.log for more details. If none of that helps, I always try creating a GenerateFlowFile processor with the text value set to something you know will work. Adjust the timer on the GenerateFlowFile processor to create a single file every 10 seconds. Use that to get the connection working. After you have exhausted all of the techniques above, figure out if the InvokeHTTP processor or PutFTP processor could be used. I hope something here helps.

Highlighted

Re: can you serialize data with NiFi?

Expert Contributor

Hi, yea its a bit tricky. Thanks for your time. Much appreciated.

Don't have an account?
Coming from Hortonworks? Activate your account here