Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Memory utilization by flume service

Highlighted

Memory utilization by flume service

Contributor

Hello All,

 

As per my knowledge, there won't be any additional physical memory usage required by flume when running apart from while doing the installation and configuration of flume.

 

Flume is jut used as channel for data ingestion. Kafka does store the data into cluster whereas flume doesn't.

 

Please confirm.

 

Thanks,

3 REPLIES 3
Highlighted

Re: Memory utilization by flume service

Cloudera Employee

Hi,

 

I assume your ceoncern is if Flume will take some space on local file system or not and not on HDFS. 

 

Flume can have file channels where the data sits on the local file system as it gets consumed by sinks. 

Also if you are using Spillable Memory Channel then data will be stored on local disk.

So we need to account for the file space required on local file system where flume channel is running.

 

 

 

But kafka or flume should not take space on HDFS. Even kafka log files are stored on local file system and not on the HDFS.

 

 

Regards

Bimal 

 

Highlighted

Re: Memory utilization by flume service

Contributor
@Bimalc,

Thanks for clarifying my doubt. I have one more doubt whether when we will be using flume, there will be physical memory usage increase or not across the flume agent host.

Thanks,

Re: Memory utilization by flume service

Master Guru
If by physical memory you mean RAM, then yes, like any other service Flume
Agents too will utilize available RAM up to its configured JVM heap size.
Don't have an account?
Coming from Hortonworks? Activate your account here