Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDFS Block size 1Gb/2GB

avatar
Rising Star

CDH enterprise 5.14.0

 

I am trying to use larger block size like 1GB/2GB.

 

In our case the files are 5GB to 14GB size and we process whole file per mapper,

 

is there any side effects to using larger block size like 1GB, 2GB? Like HDFS stability when doing replication?

1 ACCEPTED SOLUTION

avatar
Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
2 REPLIES 2

avatar
Master Collaborator

Hi @sbpothineni Why not using CombineFileInputFormat?

avatar
Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login