Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why services like Kafka and Zookeeper uses Swap space?

Highlighted

Why services like Kafka and Zookeeper uses Swap space?

Explorer

I have a 5 nodes cluster running. Where on 3 nodes I have Zookeeper and all 5 has Kafka brokers.

I see that out of 5 nodes, 3 nodes have used Swap memory. May I know why? What happens If I clear the swap memory from command line? (Linux command: swapoff -a && swapon -a)

1 REPLY 1
Highlighted

Re: Why services like Kafka and Zookeeper uses Swap space?

New Contributor

For Kafka, swap space is probably safe to clear (though I wouldn't), but you should avoid Kafka using swap space. If you look at disk IO on a Kafka broker node, it should be almost all writes, read should come from page cache. Kafka was designed to be the only tenant on a node and runs best that way. This is why you will find recommendations that say Kafka should not share nodes with Zookeeper or other Hadoop components. It is not always possible to dedicate machines to Kafka, so take a look at the disk IO when Kafka is running under normal load, if it is all writes, you can probably shrink the page cache a bit so you do less/no swapping. If there are lots of reads, you may need more memory or more nodes (unless you are deliberately and routinely reading topics from the beginning, in which case disk reads are unavoidable).

Can't help you with the zookeeper, I've never had reason to dig into zookeeper's internals, it has always just worked.

Don't have an account?
Coming from Hortonworks? Activate your account here