can you serialize data with NiFi?

[#1]

Hi, I am trying to send a list to a specific port with NiFi. However, i am getting issues when the server receives the list. I think an option is to serialize the list before sending it. Is this possible with Nifi or should i use my own code with ExecuteStreamCommand processor? Thank you

[#2]

@Roger Young ,

Which processors are you using? Sending data to a port probably involves using the HandleHTTPRequest processor.

How is the "list" being generated? Is it coming from a file, HTTPRequest, email, or something else?

[#3]

Hi @Nathan Maynes. The list is coming from a file. I am sending it to a python server on the local machine via a PutTcp processor. But every now and then the python server gives an error.

[#4]

Can you describe your intended flow in more detail? Choosing the PutTcp processor to send files is alright for low volume workloads so I would guess you are attempting to have a flow move files from one machine to another, process the file by "record"(I use the term in a very general sense to mean any data that are meant to be kept together), then do something else with the individual records. That sounds much easier with a GetFile, SplitFile, UpdateAttribute type of workflow.

[#5]

Hi @Nathan Maynes

Im using GetFile and SplitFile to take the rows of data from the file (which gives me a flowfile consisting of 30 features) then set a control rate of 1 flowfile per second before sending it to PutTCP processor. I am trying to emulate a device sending data in real time to my server on localhost, where i perform processing on the incoming flowfile. But the server gives an error which seems to be related to the format of the incoming flowfile.

[#6]

This sounds like a fun troubleshooting session~!

First, I would double check you are sending the correct delimiter characters and that the file body does not contain them. You can do this by routing the failure relationship to an inactive processor. Examine the queue list and check out some of the FlowFiles in data provenance. Try checking the logs in {path-to}/{nifi}/logs/nifi-app.log for more details. If none of that helps, I always try creating a GenerateFlowFile processor with the text value set to something you know will work. Adjust the timer on the GenerateFlowFile processor to create a single file every 10 seconds. Use that to get the connection working. After you have exhausted all of the techniques above, figure out if the InvokeHTTP processor or PutFTP processor could be used. I hope something here helps.

[#7]

Hi, yea its a bit tricky. Thanks for your time. Much appreciated.