Connecting to TCP port in NiFi

[#1]

@Pierre Villard Simple question but somewhat related to this post. I have a TCP port open on my server and able to listen on that port using a separate listener tool. I can ping that port with a message from a separate machine and see the message come through on that port so I know the port is open and receiving messages. Now, in Nifi I have a simple GetTCP processor set for that same port on the server but for some reason I am unable get any data to come through and get the following errors:

018-05-31 16:50:33,255 ERROR [pool-26-thread-2] o.a.n.processors.gettcp.ReceivingClient Exceeded wait time to connect. Possible deadlock, please report!. Interrupting. 2018-05-31 16:50:35,278 ERROR [pool-26-thread-2] o.a.n.processors.gettcp.ReceivingClient Failed to connect to secondary endpoint.

I am bit confused at this point, should I be able to accept data coming in from a separate machine over TCP inside of NiFi? Ultimately I need to do this with Consume Kafka but had no luck so decided to just test TCP.

Thanks, Jason

[#2 · Accepted solution]

@Jason Sphar

The GetTCP processor is designed to send data to a target TCP port. It sounds like you are trying to recieve data via TCP port in to NiFi?

-

If that is the case, you want to be using a ListenTCP processor instead that will bind to the tcp port you will be sending your data to.

-

Thanks,

Matt

[#3]

@Jason Sphar

*** Forum Tip: Please try to avoid responding to an existing answer by starting a new "Answer". Instead click "Add comment" on the answer you want to add correspondence. There is no guaranteed order to answers in the forum so discussions can get hard to follow.

-

As far as your new error goes. There is already some other service in this server already bound to that port. You can not have more then one service bind to a specific port.

-

You could use the following command to see what process id is using that port:

# netstat -nap |grep LISTEN

-

Then you can use following command to see which service is tied to that process id:

ps -ef |grep <program/process id>

-

Thanks,

Matt

[#4]

@Matt Clarke -- Thanks for the forum tip as well as the previous suggestions. I was able to get the ListenTCP working properly.

Jason

[#5]

@Matt Clarke - I am thinking I should ask this on another post but was curious what the proper kafka tool was for retrieving messages from a kafka topic, Consume or Get? Get looks like it's for older versions and I have been trying to use Consume_010 on Nifi 1.6. Maybe a compatibility issue between 1.6 and my kafka installation?

Thanks, Jason

[#6]

@Jason Sphar

Always best to start a new questions as this questions is unrelated to this question.

There are multiple versions of the processors for Kafka because each is for a different Kafka client version.

GetKafka/PutKafka <--- Kafka 0.8
ConsumeKafka/PublishKafka <--- Kafka 0.9
ConsumeKafka_0_10/PublishKafka_0_10/ConsumeKafkaRecord_0_10/PublishKafkaRecord_0_10 <--- Kafka 0.10

ConsumeKafka_0_11/PublishKafka_0_11/ConsumeKafkaRecord_0_11/PublishKafkaRecord_0_11 <--- Kafka 0.11

ConsumeKafka_1_0/PublishKafka_1_0/ConsumeKafkaRecord_1_0/PublishKafkaRecord_1_0 <--- Kafka 1.0

Thanks,

Matt

[#7]

Can we read data from Modbus TCP/IP protocol with Apache NiFi?

[#8]

Hey Matt,

Thanks for the response. Yes, I am trying to receive data on a TCP port in Nifi. I have now switched the processor to Listen TCP and specified the port number that I want to listen on. I left the LAN option blank and I am now getting some new errors listed below:

2018-05-31 18:05:58,393 ERROR [Monitor Processore Lifecycle Thread-1] o.a.nifi.processors.standard.ListenTCP ListenTCP[id=b74d9e21-0163-1000-391b-4e77cd11cf16] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.reflect.InvocationTargetException: java.lang.reflect.InvocationTargetException java.lang.reflect.InvocationTargetException: null

Caused by: java.net.BindException: Address already in use: bind

We have tested that the port is listening using netstat commands and it shows that only one listener is coming up so I am not sure what else could be binding the address? Is there a setting somewhere to get around this because if the data is pushing from a remote server to NiFi on this port wouldn't it essentially be connecting to that port?

Thanks, Jason