Support Questions
Find answers, ask questions, and share your expertise

Memory utilization by flume service

Contributor

Hello All,

 

As per my knowledge, there won't be any additional physical memory usage required by flume when running apart from while doing the installation and configuration of flume.

 

Flume is jut used as channel for data ingestion. Kafka does store the data into cluster whereas flume doesn't.

 

Please confirm.

 

Thanks,

3 REPLIES 3

Cloudera Employee

Hi,

 

I assume your ceoncern is if Flume will take some space on local file system or not and not on HDFS. 

 

Flume can have file channels where the data sits on the local file system as it gets consumed by sinks. 

Also if you are using Spillable Memory Channel then data will be stored on local disk.

So we need to account for the file space required on local file system where flume channel is running.

 

 

 

But kafka or flume should not take space on HDFS. Even kafka log files are stored on local file system and not on the HDFS.

 

 

Regards

Bimal 

 

Contributor
@Bimalc,

Thanks for clarifying my doubt. I have one more doubt whether when we will be using flume, there will be physical memory usage increase or not across the flume agent host.

Thanks,

Master Guru
If by physical memory you mean RAM, then yes, like any other service Flume
Agents too will utilize available RAM up to its configured JVM heap size.
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.