Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ListenTCP on a HDF cluster wants to listen to a pfSense firewall, how do I set it up

ListenTCP on a HDF cluster wants to listen to a pfSense firewall, how do I set it up

Contributor

Hi All,

I need to fetch logs from pfSense firewall and store them in hdfs, I am using ListenTCP processor but it needs a config such as "local network Interface" name, but only thing I have is the IP address and port number of the pfsense firewall, how do I configure my listenTCP to listen to this? Or do I need a different processor with config properties such as nifi.remote.input.host etc?

5 REPLIES 5

Re: ListenTCP on a HDF cluster wants to listen to a pfSense firewall, how do I set it up

Hi @dhieru singh

ListenTCP makes NiFi listen on the machine on which NiFi is running. This means that data source needs to send data to NiFi using it's IP address and port number. It's a push pattern.

I don't have big experience with pfSense but you can use syslog to export data. You can use ListenSyslog processor in NiFi to receive data. ParseSyslog can then be used to parse these logs.

https://doc.pfsense.org/index.php/Copying_Logs_to_a_Remote_Host_with_Syslog

Another option is to use MiNiFi on the firewall to collect and send data to NiFi through Site to Site

https://nifi.apache.org/minifi/

Re: ListenTCP on a HDF cluster wants to listen to a pfSense firewall, how do I set it up

Contributor

@Abdelkrim Hadjidj

Thanks for you quick resposne. I appreciate your help. The work around you suggested is good but I will not be able to do that because I am not allowed to change an exisitng running PFsense firewall runnig becuase it feeding other SIEM tool as well. Is there any other way, may be using GetTCP processor, will GetTCP processor help in this case?

I tried using Listensyslog as well however it still needs "localinterface"name

39776-question1.png

Thanks

Re: ListenTCP on a HDF cluster wants to listen to a pfSense firewall, how do I set it up

@dhieru singh

Your options will be very limited if you are not allowed to change your existing PFsense. You can either:

  • Push data to NiFi: you need to change PFsense to send data to NiFi with a supported protocol
  • Pull data with NiFi: you need to change PFsense to expose data to NiFi with a supported protocol. I don't believe that there's data exposition enabled by default
  • Use MiNiFi to get data from PFsense and send it to NiFi. This requires installing MiNiFi agent on PFsense. Is this possible?

Question: how PFsense send data to SIEM ? What use use you want to implement with NiFi?

Re: ListenTCP on a HDF cluster wants to listen to a pfSense firewall, how do I set it up

Contributor

Hi @Abdelkrim Hadjidj Thanks for the response, So I was able to talk admins of Pfsense they are now sending the logs to a load balancer in front of the nifi cluster. now they told me you can listen on it. so my load balancer IP is 192.168.88.32 and port 3421 and my nifi nodes are sitting behind the load balancer.

So in this case can I use listenTCP? , if i use it then local interface name should be of load balancr right

Re: ListenTCP on a HDF cluster wants to listen to a pfSense firewall, how do I set it up

Master Guru

@dhieru singh

I assume you are referring to the localNetwork Interface property in the ListenTCP processor.

39788-screen-shot-2017-10-12-at-121855-pm.png

You cannot configure a Listen based NiFi processor to listen on a port on a different machine from where Nifi is running.

You now have your PFsense logs being pushed to a load-balancer in front of NiFi. Where is the load-balancer configured to forward those logs?

Ideally, you would set up a ListenTCP processor that is bound to a specific port on your NiFi nodes. Then have your loadbalncer configured to forward TCP traffic in a load-balanced fashion to the listenTCP processor configured port on each of your NiFi cluster nodes.

Otherwise you are going to need to have your NiFi cluster Primary node use GetTCP to connect to your load-balancer on port 3421 to pull data sent their. This is less ideal since you don't get the benefit of a load-balanced delivery to all your NiFi cluster nodes.

Thanks,

Matt

Don't have an account?
Coming from Hortonworks? Activate your account here