Created 02-18-2018 09:47 AM
Can you please clarify few Kafka architecture question. Please answer here rather than pointing to links ( which I already did and could not understand)
i) Was it created in memory or
ii) on disk in log.dirs location
2) do consumers read the partition, that are stored in memory or from disk?
3) some of the links in google search says "Kafka purges the messages as per the retention policies --- Regardless of whether the messages has been consumed". Does this mean that consumer reads the topics from disk only and not from memory?
4) what is the relation among batch.size vs log.flush.interval.messages vs log.segment.bytes ?
4a) https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html links say, Kafka first writes data immediately to files, as soon as Log.flush.interval.messages number of messages got received.
Question is where this file is created, in memory or on disk in which location?
4b) when the log file reaches log.segment.bytes, it flushed the log file to disk.
Question is in first place where this log file is first created in memory or any other temporary location?
Created 02-19-2018 05:15 PM
Thank you very much, Harald, for addressing my questions