Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective?

[#1]

burman@Bob

[#2 · Accepted solution]

NiFi is easy at capturing logs. Why not use all technologies where they are best: NiFi to gather log data in realtime -> kafka queue -> Spark streaming analytics -> Zeppelin for spark and visualization. You could also fork NiFi to mergecontent to hdfs to keep for historical analysis.

All technologies come out-of-the-box with HDF and HDP.

[#3]

Hi Greg,

Thanks for your valuable feedback. I am quite new in this field and recently, I am trying to implement this into my company. I need one more positive feedback on one tutorial that I found as below-

"http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/"

I prefer only Hortonworks tutorial.

Best Regards,

Bob

[#4]

@Bibhas BurmanThat is an excellent tutorial for pushing log data to HDFS for historical analysis. If you want to do real-time streaming analysis here are two links that should be useful

http://hortonworks.com/hadoop-tutorial/realtime-event-processing-nifi-kafka-storm/ (ignore the storm part)

https://community.hortonworks.com/articles/44550/horses-for-courses-apache-spark-streaming-and-apac.... (integrate with the kafka part from the first link)

Since you are getting your feet wet with the technology, definitely put in some time to play around with it and build small projects before working toward your end product. And of course ... anytime you have a question along the way ask the HCC to get some guidance.

[#5]

It is very informative and helped me a lot. Thanks.

[#6]

Hi Greg,

A simple question, if someone ask me why should use Apche NiFi using Big Data technology to analyze log files? We have microsoft-logparser. What is the advantage of using Big Data technology such as HDF nifi..?