Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Application Server Log processing


Application Server Log processing


Hi All ,

I have a requirement from my client to process the application(Tomcat) server log file for a back end REST Based App server which is deployed on a cluster. Clint wants to generate "access" and "frequency" report from those data with different parameter.

My initial plan is that get those data from App server log -- push to Spark Streaming using kafka and process the data -- store those data to HIVE -- use zeppelin to get back those processed and centralized log data and generate reports as per client requirement.

But as per my knowledge Kafka does not any feature which can read data from log file and post them in Kafka broker by its own , in that case we have write a scheduler job process which will read the log time to time and send them in Kafka broker , which I do not prefer to do, as in that case it will not be a real time and there can be synchronization issue which we have to bother about as we have 4 instances of application server.

Another option, I think we have in this case is Apache Flume.

Can any one suggest me which one would be better approach in this case or if in Kafka, we have any process to read data from log file by its own and what are the advantage or disadvantages we can have in feature in both the cases?

I guess another option is Flume + kakfa together , but I can not speculate much what will happen as I have almost no knowledge about flume.

Any help will be highly appreciated...... :-)

Thanks a lot ....


Re: Application Server Log processing

Rising Star

@Biswajit Chakraborty this is the perfect problem for Apache NiFI (HDF) to solve. Additionally, unless your use case requires only a few things to be computed, I would recommend building a NiFi -> Kafka -> Spark or Storm -> Solr -> Silk pipeline making it more extensible.

Don't have an account?
Coming from Hortonworks? Activate your account here