Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

what is the preferred file system for kafka and data node

what is the preferred file system for kafka and data node

hi all


what is the preferred file system for data disks for kafka and data-node machines


XFS or EXT4 ?



note - for now we are using - ext4

Michael-Bronson
4 REPLIES 4
Highlighted

Re: what is the preferred file system for kafka and data node

Super Mentor

@Michael Bronson

I do not think there is any specific recommendation for HDFS/Kafka in terms of XFS/EXT4 but from OS side there is a good comparison and description is available here:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration...

.

Some generic filesystem recommendations from hwx docs: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_cluster-planning/content/ch_partitioning...
.

HCC Thread: https://community.hortonworks.com/questions/5086/ext4-vs-xfs-filesystem-survey-of-popularity.html

Re: what is the preferred file system for kafka and data node

@Jay - I see in the second link something that I not agree , link say - " Swap: Size 2X system memory " , as I know we need to disable the swap on all datanode and kafka machines !

Michael-Bronson

Re: what is the preferred file system for kafka and data node

Super Mentor

@Michael Bronson

I feel you are right.

I also see similar threads on HCC for the Swap recommendations:

1. Your Old HCC discussion on Swap: https://community.hortonworks.com/questions/202215/swap-size-howrtonworks-recommendation-in-hadoop-c...
2. Other HCC discussion on Swap: https://community.hortonworks.com/questions/22548/what-is-the-hortonworks-recommendation-on-swap-usa...

Re: what is the preferred file system for kafka and data node

@Jay , since the swap is very critical , then who can update the hortonworks document ? , I think many people that saw that maybe do the mistake with the swap , so it must be update soon

Michael-Bronson