I need to fetch logs from pfSense firewall and store them in hdfs, I am using ListenTCP processor but it needs a config such as "local network Interface" name, but only thing I have is the IP address and port number of the pfsense firewall, how do I configure my listenTCP to listen to this? Or do I need a different processor with config properties such as nifi.remote.input.host etc?
ListenTCP makes NiFi listen on the machine on which NiFi is running. This means that data source needs to send data to NiFi using it's IP address and port number. It's a push pattern.
I don't have big experience with pfSense but you can use syslog to export data. You can use ListenSyslog processor in NiFi to receive data. ParseSyslog can then be used to parse these logs.
Another option is to use MiNiFi on the firewall to collect and send data to NiFi through Site to Site
Thanks for you quick resposne. I appreciate your help. The work around you suggested is good but I will not be able to do that because I am not allowed to change an exisitng running PFsense firewall runnig becuase it feeding other SIEM tool as well. Is there any other way, may be using GetTCP processor, will GetTCP processor help in this case?
I tried using Listensyslog as well however it still needs "localinterface"name
Your options will be very limited if you are not allowed to change your existing PFsense. You can either:
Question: how PFsense send data to SIEM ? What use use you want to implement with NiFi?
Hi @Abdelkrim Hadjidj Thanks for the response, So I was able to talk admins of Pfsense they are now sending the logs to a load balancer in front of the nifi cluster. now they told me you can listen on it. so my load balancer IP is 192.168.88.32 and port 3421 and my nifi nodes are sitting behind the load balancer.
So in this case can I use listenTCP? , if i use it then local interface name should be of load balancr right
I assume you are referring to the localNetwork Interface property in the ListenTCP processor.
You cannot configure a Listen based NiFi processor to listen on a port on a different machine from where Nifi is running.
You now have your PFsense logs being pushed to a load-balancer in front of NiFi. Where is the load-balancer configured to forward those logs?
Ideally, you would set up a ListenTCP processor that is bound to a specific port on your NiFi nodes. Then have your loadbalncer configured to forward TCP traffic in a load-balanced fashion to the listenTCP processor configured port on each of your NiFi cluster nodes.
Otherwise you are going to need to have your NiFi cluster Primary node use GetTCP to connect to your load-balancer on port 3421 to pull data sent their. This is less ideal since you don't get the benefit of a load-balanced delivery to all your NiFi cluster nodes.