Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDF be used to feed Logstash?

avatar
Expert Contributor

I am thinking about setting up a Logstash infrastructure to monitor my system. (It happens to be Hortonworks HDP Hadoop cluster, but assume it isn't). So I have various things which generate logs and I want to transfer these logs outside my system to a new system - such as ElasticSearch inside Logstash. And I want to do this securely.

I don't really want Flume for this as there are better tools.

Now I might use Logstash forwarders - which most recently seems to be a new system called "Beats" - in particular FileBeat. However I would prefer to use Apache NiFi because of its security reputation. I would like to use HDF as I am a Hortonworks Partner and we are already using HDP.

Can anyone say:

"Yes this makes sense", "Yes, I have done it", "You need to read URL blah blah blah"?

Or have I got the wrong end of the stick?

PS I know that Ambari Metrics moves operational logs from the Hadoop cluster into the HDFS system - this is separate from that.

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Guru

Advantage of using HDF here is that you can do any preprocessing/filtering on your logs before you put into ElasticSearch. This is one of the common usecase where logs are preprocessed before putting into a system like Splunk/Logstash.

avatar
Expert Contributor

Thanks. That is helpful

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

Thanks people. That is very helpful. It sounds like I have some learning to do 🙂