question Re: Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective? in Archives of Support Questions (Read Only)

Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective?

bibhas_burman — Tue, 11 Oct 2016 02:46:43 GMT

burman@Bob

Re: Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective?

gkeys — Tue, 11 Oct 2016 03:27:21 GMT

NiFi is easy at capturing logs. Why not use all technologies where they are best: NiFi to gather log data in realtime -> kafka queue -> Spark streaming analytics -> Zeppelin for spark and visualization. You could also fork NiFi to mergecontent to hdfs to keep for historical analysis.

All technologies come out-of-the-box with HDF and HDP.

Re: Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective?

bibhas_burman — Tue, 11 Oct 2016 04:37:07 GMT

Hi Greg,

Thanks for your valuable feedback. I am quite new in this field and recently, I am trying to implement this into my company. I need one more positive feedback on one tutorial that I found as below-

"http://hortonworks.com/hadoop-tutorial/how-to-refine-and-visualize-server-log-data/"

I prefer only Hortonworks tutorial.

Best Regards,

Bob

Re: Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective?

gkeys — Tue, 11 Oct 2016 06:16:28 GMT

@Bibhas BurmanThat is an excellent tutorial for pushing log data to HDFS for historical analysis. If you want to do real-time streaming analysis here are two links that should be useful

http://hortonworks.com/hadoop-tutorial/realtime-event-processing-nifi-kafka-storm/ (ignore the storm part)

https://community.hortonworks.com/articles/44550/horses-for-courses-apache-spark-streaming-and-apac.html (integrate with the kafka part from the first link)

Since you are getting your feet wet with the technology, definitely put in some time to play around with it and build small projects before working toward your end product. And of course ... anytime you have a question along the way ask the HCC to get some guidance.

Re: Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective?

bibhas_burman — Fri, 14 Oct 2016 02:34:49 GMT

It is very informative and helped me a lot. Thanks.

Re: Q1. Which is the best option in IT industry to analyze Web Server Log : 1. HDF NiFi with Apache Zeppelin OR 2. new Zeppelin notebook using Spark? Q2. Which is cost effective?

bibhas_burman — Fri, 14 Oct 2016 02:39:37 GMT

Hi Greg,

A simple question, if someone ask me why should use Apche NiFi using Big Data technology to analyze log files? We have microsoft-logparser. What is the advantage of using Big Data technology such as HDF nifi..?