How to send Windows event log to HCP ?

[#1]

Hi

I want to send Windows event log to HCP ( with any agent like winlogbeats or etc ) but I don't know how to do this ? can you provide solution ?

Thanks

[#2 · Accepted solution]

Hi @haco fayik,

as a starting point you need to push data into a parser specific Kafka topic (you can call the topic "windows-event-log"), and configure a parser in the Metron Management UI and start it. In the parser configuration you configure Metron, from which Kafka topic the messages are picked up ("windows-event-log" in our case) and how to parse the incoming messages.

NiFi is a great tool to collect data from various sources and push it into Kafka.

Maybe my article helps you: https://datahovel.com/2018/07/18/how-to-onboard-a-new-data-source-in-apache-metron/

If you have more specific questions, don't hesitate to ask!

[#3]

hi @Stefan Kupstaitis-Dunkler,

Thank you so much for your answer ,

if I have 5 windows server and workstation , I should install nifi on each host or I can use one nifi server for all hosts ?

How to send data ( event log) to nifi ?

[#4]

@haco fayik

There's many ways to do this. You should probably search this community in the NiFi section or get familiar with NiFi in general.

However, as a a short overview, the most common cases for Metron ingestion, I'm encountering in the field are:

• your sources are pushing the message to a syslog server. You can configure your syslog server to push data to your NiFi instance over TCP or UDP. In this case you'd need a "ListenSyslog" processor and a "PublishKafka" processor.
• you already have a log forwarder capable of pushing data to Kafka (winlogbeats😞 https://www.elastic.co/guide/en/beats/winlogbeat/current/configuring-output.html . In this case you won't need NiFi, if you are comfortable using winlogbeats.
• You install MiNiFi on all servers to act as a simple log forwarder over tcp. You'd send those packets to a NiFi instance/cluster (similar to the Syslog approach), receive them via "ListenTcp" processor and push your messages into Kafka using the "PublishKafka" processor. You could also send data directly into Kafka from MiNiFi.

Note: If your Kafka cluster is secured with Kerberos, this might influence your choice.

[#5]

thank you very much @Stefan Kupstaitis-Dunkler

[#6]

Hi @Stefan Kupstaitis-Dunkler

I Installed winlogbeats on Windows workstation with below config :

output.logstash:
  hosts: ["nifi.node.srv:5098"]

and I use this nifi processors to stream event to metron

listenbeats config :

Publishkafka cofig :

Nifi Data provenance in publishkafka processor :

and I create sensor in Management UI with logstash parser and winlogtop topic ( kafka) . now I can't see any log data in alert UI . what's problem ?

Thanks

[#7]

Hi @haco fayik

That looks great. Sounds like you got around the initial problem of ingesting data into Metron.

There could be multiple reasons, e.g. parser, enrichment and indexing topologies not running or being misconfigured.

Would you create a new question for this and provide more details, such as worker logs of those topologies?

Would you also mark the answer that helped you most solve the ingest problem as "Best Answer"?

thanks!

[#8]

Thanks @Stefan Kupstaitis-Dunkler,

I marked best answer and I will create a new question for this problem . Can you provide location of these log file?

I confused that Can I use metron for Collect windows and linux hosts and network devices log for security purpose ? ( Threat detection and etc)

Please accept my thanks for your helps