Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to send Windows event log to HCP ?

Solved Go to solution

How to send Windows event log to HCP ?

New Contributor

Hi


I want to send Windows event log to HCP ( with any agent like winlogbeats or etc ) but I don't know how to do this ? can you provide solution ?

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to send Windows event log to HCP ?

Contributor

Hi @haco fayik,

as a starting point you need to push data into a parser specific Kafka topic (you can call the topic "windows-event-log"), and configure a parser in the Metron Management UI and start it. In the parser configuration you configure Metron, from which Kafka topic the messages are picked up ("windows-event-log" in our case) and how to parse the incoming messages.

NiFi is a great tool to collect data from various sources and push it into Kafka.

Maybe my article helps you: https://datahovel.com/2018/07/18/how-to-onboard-a-new-data-source-in-apache-metron/

If you have more specific questions, don't hesitate to ask!

7 REPLIES 7

Re: How to send Windows event log to HCP ?

Contributor

Hi @haco fayik,

as a starting point you need to push data into a parser specific Kafka topic (you can call the topic "windows-event-log"), and configure a parser in the Metron Management UI and start it. In the parser configuration you configure Metron, from which Kafka topic the messages are picked up ("windows-event-log" in our case) and how to parse the incoming messages.

NiFi is a great tool to collect data from various sources and push it into Kafka.

Maybe my article helps you: https://datahovel.com/2018/07/18/how-to-onboard-a-new-data-source-in-apache-metron/

If you have more specific questions, don't hesitate to ask!

Re: How to send Windows event log to HCP ?

New Contributor

hi @Stefan Kupstaitis-Dunkler,

Thank you so much for your answer ,

if I have 5 windows server and workstation , I should install nifi on each host or I can use one nifi server for all hosts ?

How to send data ( event log) to nifi ?

Highlighted

Re: How to send Windows event log to HCP ?

Contributor

@haco fayik

There's many ways to do this. You should probably search this community in the NiFi section or get familiar with NiFi in general.

However, as a a short overview, the most common cases for Metron ingestion, I'm encountering in the field are:

  • your sources are pushing the message to a syslog server. You can configure your syslog server to push data to your NiFi instance over TCP or UDP. In this case you'd need a "ListenSyslog" processor and a "PublishKafka" processor.
  • you already have a log forwarder capable of pushing data to Kafka (winlogbeats): https://www.elastic.co/guide/en/beats/winlogbeat/current/configuring-output.html . In this case you won't need NiFi, if you are comfortable using winlogbeats.
  • You install MiNiFi on all servers to act as a simple log forwarder over tcp. You'd send those packets to a NiFi instance/cluster (similar to the Syslog approach), receive them via "ListenTcp" processor and push your messages into Kafka using the "PublishKafka" processor. You could also send data directly into Kafka from MiNiFi.

Note: If your Kafka cluster is secured with Kerberos, this might influence your choice.

Re: How to send Windows event log to HCP ?

New Contributor

thank you very much @Stefan Kupstaitis-Dunkler

Re: How to send Windows event log to HCP ?

New Contributor

Hi @Stefan Kupstaitis-Dunkler

I Installed winlogbeats on Windows workstation with below config :

output.logstash:
  hosts: ["nifi.node.srv:5098"]

and I use this nifi processors to stream event to metron

97533-nifi1.png

listenbeats config :

97534-nifi2.png

Publishkafka cofig :

97535-nifi3.png

Nifi Data provenance in publishkafka processor :

97536-nifi4.png

and I create sensor in Management UI with logstash parser and winlogtop topic ( kafka) . now I can't see any log data in alert UI . what's problem ?

Thanks

Re: How to send Windows event log to HCP ?

Contributor

Hi @haco fayik

That looks great. Sounds like you got around the initial problem of ingesting data into Metron.

There could be multiple reasons, e.g. parser, enrichment and indexing topologies not running or being misconfigured.

Would you create a new question for this and provide more details, such as worker logs of those topologies?

Would you also mark the answer that helped you most solve the ingest problem as "Best Answer"?

thanks!

Re: How to send Windows event log to HCP ?

New Contributor

Thanks @Stefan Kupstaitis-Dunkler,

I marked best answer and I will create a new question for this problem . Can you provide location of these log file?

I confused that Can I use metron for Collect windows and linux hosts and network devices log for security purpose ? ( Threat detection and etc)

Please accept my thanks for your helps